We want to learn things from any idea, test, change, upgrade or (heaven forbid) outage in production..
How do we do this and, how do we do it quickly?
I shouldn’t have to say this.. If you make one change in a system and it performs better or worse; you have learned something, almost immediately. If you make more than one change in a system and it changes the performance or causes an outage, you likely haven’t learned very much at all; or it will take much longer to find out.
Any test should be testing one variable while eliminating other variables [not under test] as much as possible.
This should be common sense, yet many organizations still do system upgrades with dozens of changes with each release. People still do tests where they adjust a dozen knobs to see if performance improves and (and attempt to) make judgement calls about the changes.
Any change in production should be a learning experience. You can’t do that if you’re using a shotgun..