Ask the right questions, listen carefully and make sure that you’re not hearing just what you want to hear..
Some thoughts about handling critical system issues at scale..
We called our albino squirrel in the backyard, “Snowflake”..
As an SRE, I’m very fortunate to have had training as a pilot. There are many similarities to system operations..
Building interactive commands that uses editing history and tab completion can be easy in bash and serve as a wrapper for automating tasks.
Not just “Must Have”, but “Must Read!”. A new book has been released and is available, free to download for a short time.
Most of the operations/engineering folks I’ve come into contact with will proclaim to be “vi” people and yet, when I watch them edit a file I cringe..
When we talk of CI/CD we’re often referring to Continuous Integration and Delivery while Optimization refers to Services/Systems. What I’d like to discuss is Constant Improvement/Continuous Development and Self-Optimization..
In a previous article, “There’s Always a Problem”, I described situations that can arise with the “Engineering vs. Operations” old way. The new way is a DevOps culture..
I keep a large framed photo of this on the wall in my office to remind me what can happen when things go “off the rails”..
A colleague of mine once posted a hiring question to ask prospective developers: “What is the least significant 10 digits of the series: .. ?”
In previous posts, I’ve emphasized that averages are particularly bad at characterizing most things that you might be looking for. However, storing aggregated data of any type can limit your ability to analyze data later.
Many people use awk for one-liners; picking out fields from logs, doing pattern matching. It’s capable of so much more however. IMO, the “littleness” of the language is one of it’s strengths.
This article has a link to a simple script I’ve used for over a decade to detect corrupted files. It will detect and report on files that have changed, been added, deleted or possibly moved within the same directory structure.
Your systems have drives set up in RAID configurations and besides, you have data copied to redundant systems and backups, right? Safe? Maybe not. I recently found corruption in a quarter of a million files that had not previously been detected, for years!
Do you have insatiable curiosity and are driven by a relentless pursuit of the truth? You might make a great problem solver, but be careful how you deal with your findings!
If you’ve been around systems long enough, you know that opportunity for performance gains goes up dramatically, the further up the stack you look..
For years I’ve done most of my log scraping and analysis with the usual suspects; bash, sed, awk, perl even. The log scraping still uses those tools, but lately I’ve been toying around with “R” for the analysis.
..unless you’re dealing with baseball. When dealing with systems, many of us think “Average” is a measure of “Typical” or “Normal“. Many systems people will also use averages to look for “Abnormal“. However, average (or mean) doesn’t represent either “normal” or “abnormal” very well..
Many of us have dealt with making changes in production environments, possibly against hundreds or thousands of systems and, we’d like to know how the change impacted performance. It was with this in mind that I eagerly read through the paper describing WSMeter.
Just for fun, how many combinations of months are there where Friday falls on the 13th? This one-liner will print out a table of month combinations along with the years for a given range.
While working at RIM, I had the privilege of working with some brilliant engineers. During that time I developed a few of the techniques that I’ll be describing; the EPD (Event-Pair-Difference) graph described in my previous post and the EPL (Event-Pair-Latency) Dotplot are a few of them.
System failures are often not black and white, but shades of grey (gray?).. Detecting and alerting on “performance-challenged” system components are a lot more difficult than detecting black or white (catastrophic failures). The metrics used are usually of the “time vs. latency” or “time vs. event count” variety, often aggregated and, often by using averages. …