If you’ve been around systems long enough, you know that opportunity for performance gains goes up dramatically, the further up the stack you look..
Engineering will often suggest adding more hardware to increase performance. This will be somewhat less than a linear improvement however. To double the performance, double the hardware and hope that there isn’t too much overhead involved. There are other alternatives, depending on what the bottleneck is.
Certainly, if the bottleneck is found to be “spinning rust”, moving it to SSD would help considerably. Memory/Swap issues? Add more memory (not swap). There could be some gains here, but remember that today, we are often faced with thousands of systems, not dozens like we were many years ago, so the cost may be significant.
The OS level has many knobs that you may be able to tweak. Typical improvements here range in the single digits of percentage improvement. Significant and, while there are exceptions (i.e. if the OS was badly optimized in the first place), it’s rare to get a huge increase in performance. Nevertheless, when dealing with thousands of systems, cost savings can scale up too.
Middle tier includes things like databases. We start getting into the realm of where more significant improvements can be made with some thoughtful tuning. Double digit percentage improvements in performance are common. (If you’re dealing with MySQL issues, I highly recommend “High Performance MySQL”[1]. Slightly out of date, but excellent.)
Finally, we have the application level. Huge performance gains are often possible here, often measured in dozens of times (thousands of percent) increase in performance. Problems found can be things like: poor algorithms, locking issues, using database indexes improperly resulting in table scanning of billions of records, etc.
(Actually, at the very top of the stack, we have the “User“. Sometimes processes can be optimized right out if no longer required; or user procedures changed, users educated, etc.)
Be wary of finding these issues however. If you are in the operations group, you may not make any friends with your software engineering colleagues. You’ll need a strong operations management that can back you up. That’s the subject of my next article, There’s Always a Problem.
[1] High Performance MySQL, 3rd Edition; Peter Zaitsev, Vadim Tkachenko, Baron Schwartz, O’Reilly Media; http://shop.oreilly.com/product/0636920022343.do