I’m a big believer in “wrapper scripts”.
Category: SRE
K8S Tips and Scripts #1
I’ve often come across situations where pods won’t die properly. Either the underlying node is in a strange state, temporary networking issues, whatever. If a node has gone south, pods will often stick in an “Evicted” state for example.
Where Are We Going?
“Where are we going?” my wife asks as we leave the runway on her first flight with me after I received my pilot license. Continue reading “Where Are We Going?”
Finding the Information You Need
Nothing sucks enthusiasm for a job faster than a frustrating search for information you need to do the job that turns up empty or irrelevant. Even worse is when the information you find is outdated or even lies to you!
Thoughts on Documentation
When I worked at RIM (Blackberry), we had the luxury of having technical writers on staff. They were brilliant at yanking information out of engineering and organizing it into cohesive sets of documentation. Not all companies or departments can afford a good technical writers on staff, so we all need to pitch in and do it. What constitutes good technical writing?
K8S Tips and Scripts #0
It’s time to start writing again.. It’s been awhile..
I’ll start by sharing some tips I’ve learned and scripts I’ve written for dealing with Kubernetes.
Implementing Service Level Objectives
I picked up a new book recently that is a great companion to the other SRE books published by O’Reilly. Continue reading “Implementing Service Level Objectives”
NOTAM for SREs
In aviation, NOTAMs are “Notices to Airmen” for conditions that are generally temporary and hence not information published in the usual places. Continue reading “NOTAM for SREs”
Command Line Interface Guidelines
Anyone who knows me, knows that I am most comfortable and at home on the unix/linux command line. Continue reading “Command Line Interface Guidelines”
The Tail at Scale Approximation
This article demonstrates a quick and easy approximation for the probability formulae which I described in two previous articles. Continue reading “The Tail at Scale Approximation”
The Tail at Scale Revisited
My last article discussed some of the missing math related to setting back-end objectives. This article presents a chart which is useful in understanding the relationship to the user experience and we examine ways to dramatically improve the overall performance. Continue reading “The Tail at Scale Revisited”
The Tail at Scale
The landmark “Tail at Scale”[1] article was missing some of the math. We’re diving into it a bit here to show how the math can be used in setting objectives for latency budgets in back end systems. Continue reading “The Tail at Scale”
What is SRE?
The current state of confusion around what a “Site Reliability Engineer” (SRE) role is..
Continue reading “What is SRE?”
BPF Performance Tools
BPF is one of the Swiss Army Knife tools for Performance Engineering on Linux. Continue reading “BPF Performance Tools”
Event Logs and A.I.
Many companies in the logging/monitoring space will try to sell you on AI and ML (Artificial Intelligence and Machine Learning) to find abnormal. Continue reading “Event Logs and A.I.”
Event Logs and K.I.S.S.
I’ve worked with event logs for, well, decades. There are quite a few companies that offer services for managing logs and, afaik, only a few doing it right. Continue reading “Event Logs and K.I.S.S.”
SPOFs and Partial Panel
In both aviation and systems we build in redundancies wherever practical to avoid unpleasantness when components or subsystems fail. Continue reading “SPOFs and Partial Panel”
Traffic At 2 O’clock!
Up in the air, your eyes can’t be everywhere, all the time. You’re trained to scan the skies for “traffic” (other flying machines) as well as scanning instrumentation in the cockpit. Continue reading “Traffic At 2 O’clock!”
Own It !!
We were heading back from the practice area to the airport. I didn’t have my pilot license yet and my instructor says: “Push the throttle to Rental Speed!”. Continue reading “Own It !!”
Reading Week #4
Monitoring the SRE Golden Signals, an excellent overview by Steve Mushero.. Continue reading “Reading Week #4”
Reading Week #3
Here are some interesting reads if you’re fortunate in having some extra time off this Holiday Season.. Continue reading “Reading Week #3”
Reading Week #2
First of all, Merry Christmas if you celebrate it, Happy Holidays if you don’t! This week’s interesting read is about a subject I love.. Continue reading “Reading Week #2”
Reading Week #1
I’d like to start a new series of articles based on interesting articles to read for the week.. Continue reading “Reading Week #1”
Checklists and Runbooks
We’ve been flying planes much longer than we’ve been running systems in production, so it might be instructive to learn what we can from our fellow aviators.. Continue reading “Checklists and Runbooks”
A Steal on O’Reilly DevOps/SRE Books!
This is a limited time deal on 15 O’Reilly books for $15. Go. Buy. Right. Now! Continue reading “A Steal on O’Reilly DevOps/SRE Books!”
All Day DevOps 2018
I was lucky to catch most of the SRE track and Keynote speakers with the All Day DevOps event this year. Fortunately, if you missed it or want to watch some of the other tracks, the videos have been made available.
One Thing At A Time..
We want to learn things from any idea, test, change, upgrade or (heaven forbid) outage in production..
Seeking SRE
This book just arrived this morning and I’m just through the chapter on building SRE teams. Continue reading “Seeking SRE”
Don’t Panic!
Some thoughts about handling critical system issues at scale.. Continue reading “Don’t Panic!”
Snowflakes
We called our albino squirrel in the backyard, “Snowflake”..