One Thing At A Time..

We want to learn things from any idea, test, change, upgrade or (heaven forbid) outage in production..

How do we do this and, how do we do it quickly?

I shouldn’t have to say this..  If you make one change in a system and it performs better or worse; you have learned something, almost immediately.  If you make more than one change in a system and it changes the performance or causes an outage, you likely haven’t learned very much at all; or it will take much longer to find out.

Any test should be testing one variable while eliminating other variables [not under test] as much as possible.

This should be common sense, yet many organizations still do system upgrades with dozens of changes with each release.  People still do tests where they adjust a dozen knobs to see if performance improves and (and attempt to) make judgement calls about the changes.

Any change in production should be a learning experience. You can’t do that if you’re using a shotgun..