Performance Matters (and Other Tidbits from the Trenches)
The first article ended on the catch: a service catalog is only as good as its data, and the people who want it most — SREs — aren’t the people who can keep it accurate. The engineering teams are.
This one is about that part. Standing up Backstage was real work, but it was the easy part. The hard part was getting engineering teams to care about a YAML file in their repo.
If you’ve spent any time on call, you’ve asked every one of these questions, usually at the worst possible moment:
Most graphs we see as SREs are time based along the X axis, whether it is request rates, error rates, latencies or some other parameter along the Yaxis.
Pilots have a specific term for moving from one airplane to another: a “type conversion”. Not “learning to fly” again — you already know how to fly. It’s the process of taking everything you already know and re-mapping it onto a new machine that does the same job with a different cockpit.
Continue reading “Type Conversion”
I’ve spent most of my career in the “in-between” role — the one that sits between software engineering and operations, making sure the systems that carry the weight of the business stay up, stay fast, and stay honest about what they’re actually doing. Listen, follow the data, don’t do anything you can’t undo quickly. That’s been the job for a couple of decades.
In the last year, the job changed more than it did in the previous ten.
Continue reading “AI and SRE”
openssl is a powerful utility, but how often do you need to look up the options when you want to use it?
During the recent Holidays I visited some friends who are avid Scrabble players. After the visit, I thought it would be fun to write a simple anagram script.
The next time Friday 13th falls in January is in 2034.. Good luck!
I’m a big believer in “wrapper scripts”.
I’ve often come across situations where pods won’t die properly. Either the underlying node is in a strange state, temporary networking issues, whatever. If a node has gone south, pods will often stick in an “Evicted” state for example.
“Where are we going?” my wife asks as we leave the runway on her first flight with me after I received my pilot license. Continue reading “Where Are We Going?”
Nothing sucks enthusiasm for a job faster than a frustrating search for information you need to do the job that turns up empty or irrelevant. Even worse is when the information you find is outdated or even lies to you!
When I worked at RIM (Blackberry), we had the luxury of having technical writers on staff. They were brilliant at yanking information out of engineering and organizing it into cohesive sets of documentation. Not all companies or departments can afford a good technical writers on staff, so we all need to pitch in and do it. What constitutes good technical writing?
It’s time to start writing again.. It’s been awhile..
I’ll start by sharing some tips I’ve learned and scripts I’ve written for dealing with Kubernetes.
I picked up a new book recently that is a great companion to the other SRE books published by O’Reilly. Continue reading “Implementing Service Level Objectives”
In aviation, NOTAMs are “Notices to Airmen” for conditions that are generally temporary and hence not information published in the usual places. Continue reading “NOTAM for SREs”
Anyone who knows me, knows that I am most comfortable and at home on the unix/linux command line. Continue reading “Command Line Interface Guidelines”
This article demonstrates a quick and easy approximation for the probability formulae which I described in two previous articles. Continue reading “The Tail at Scale Approximation”
I love AWK and I’ve written about it before… Continue reading “Riddler: Can you solve the not-so-corn maze?”
My last article discussed some of the missing math related to setting back-end objectives. This article presents a chart which is useful in understanding the relationship to the user experience and we examine ways to dramatically improve the overall performance. Continue reading “The Tail at Scale Revisited”
The landmark “Tail at Scale”[1] article was missing some of the math. We’re diving into it a bit here to show how the math can be used in setting objectives for latency budgets in back end systems. Continue reading “The Tail at Scale”
This was an interesting question, so I thought I’d share it here.. Continue reading “Hiring Questions, Problem 3”
Many companies and individuals are contemplating remote work now. Embrace it! Continue reading “Remote Work”
The current state of confusion around what a “Site Reliability Engineer” (SRE) role is..
Continue reading “What is SRE?”
BPF is one of the Swiss Army Knife tools for Performance Engineering on Linux. Continue reading “BPF Performance Tools”
Many companies in the logging/monitoring space will try to sell you on AI and ML (Artificial Intelligence and Machine Learning) to find abnormal. Continue reading “Event Logs and A.I.”
I’ve worked with event logs for, well, decades. There are quite a few companies that offer services for managing logs and, afaik, only a few doing it right. Continue reading “Event Logs and K.I.S.S.”
In both aviation and systems we build in redundancies wherever practical to avoid unpleasantness when components or subsystems fail. Continue reading “SPOFs and Partial Panel”
Up in the air, your eyes can’t be everywhere, all the time. You’re trained to scan the skies for “traffic” (other flying machines) as well as scanning instrumentation in the cockpit. Continue reading “Traffic At 2 O’clock!”
We were heading back from the practice area to the airport. I didn’t have my pilot license yet and my instructor says: “Push the throttle to Rental Speed!”. Continue reading “Own It !!”