Generated by All in One SEO v5.0.0.1, this is an llms.txt file, used by LLMs to index the site.

# bill duncan's blog

Performance Matters (and Other Tidbits from the Trenches)

## Sitemaps

- [XML Sitemap](https://billduncan.org/sitemap.xml): Contains all public & indexable URLs for this website.

## Posts

- [AI and SRE](https://billduncan.org/ai-and-sre/) - AI and SRE: The Same Force Cuts Both Ways I've spent most of my career in the "in-between" role -- the one that sits between software engineering and operations, making sure the systems that carry the weight of the business stay up, stay fast, and stay honest about what they're actually doing. Listen, follow the
- [openssl wrapper script](https://billduncan.org/openssl-wrapper-script/) - openssl is a powerful utility, but how often do you need to look up the options when you want to use it? The 80/20 rule.. This script will do what you want most of the time (for me anyway), without hitting the manpage. It spits out most of the relevant information you want, whether you Useful simple script for using openssl to check certificate information
- [The Tail at Scale](https://billduncan.org/the-tail-at-scale/) - Some of the math behind nines used to establish back end service level objectives (SLO).
- [Anagrams for Fun](https://billduncan.org/anagrams-for-fun/) - During the recent Holidays I visited some friends who are avid Scrabble players. After the visit, I thought it would be fun to write a simple anagram script. Not much to this. It uses whatever dictionary of words you might have around, filters out any that have an apostrophe, generates a ``signature'' for each word
- [Good Luck and Happy New Year!](https://billduncan.org/good-luck-and-happy-new-year/) - The next time Friday 13th falls in January is in 2034.. Good luck! Friday the 13th One Liner 1 April July 2001 2007 2018 2029 2035 2046 2 August 2010 2021 2027 2038 2049 3 February August 2004 2032 4 February March November 2009 2015 2026 2037 2043 5 January April July 2012 2040
- [What is SRE?](https://billduncan.org/what-is-sre/) - About the confusion around what a ``Site Reliability Engineer'' (SRE) role is.
- [Operations in the Cloud](https://billduncan.org/operations-in-the-cloud/) - The importance of instrumenting cloud operations for situational awareness. Intuitions are not enough.
- [K8S Tips and Scripts #2](https://billduncan.org/k8s-tips-and-scripts-2/) - I'm a big believer in ``wrapper scripts''. Wrapper Scripts If available commands don't do exactly what you want, don't reinvent another wheel, bend what you have available to do what you want. Let the available tools do the ``heavy lifting'' while you just adjust to taste. A great example of this is looking at node
- [K8S Tips and Scripts #1](https://billduncan.org/k8s-tips-and-scripts-1/) - I've often come across situations where pods won't die properly. Either the underlying node is in a strange state, temporary networking issues, whatever. If a node has gone south, pods will often stick in an "Evicted" state for example. Delete All Misbehaving Pods There can also be events which will leave hundreds of pods in
- [Where Are We Going?](https://billduncan.org/where-are-we-going/) - ``Where are we going?'' my wife asks as we leave the runway on her first flight with me after I received my pilot license.``Up!'' I responded. She looked disappointed. She was thinking that we'd head to another city airport, perhaps for a $100 hamburger. She was unaware of the planning involved to fly to another
- [Checklists and Runbooks](https://billduncan.org/checklists-and-runbooks/) - Checklists for aviators predates runbooks for systems, but they have similar use cases.
- [Finding the Information You Need](https://billduncan.org/finding-the-information-you-need/) - Nothing sucks enthusiasm for a job faster than a frustrating search for information you need to do the job that turns up empty or irrelevant. Even worse is when the information you find is outdated or even lies to you! How Systems Do It While looking for information I needed at work recently, I was
- [Thoughts on Documentation](https://billduncan.org/thoughts-on-documentation/) - When I worked at RIM (Blackberry), we had the luxury of having technical writers on staff. They were brilliant at yanking information out of engineering and organizing it into cohesive sets of documentation. Not all companies or departments can afford a good technical writers on staff, so we all need to pitch in and do
- [K8S Tips and Scripts #0](https://billduncan.org/k8s-tips-and-scripts-0/) - It's time to start writing again.. It's been awhile.. I'll start by sharing some tips I've learned and scripts I've written for dealing with Kubernetes. Krew Plugin Manager There are many tools like lens and rancher for dealing with Kubernetes with a ``user friendly'' UI.. Personally, I'm more of a command line guy and so
- [Implementing Service Level Objectives](https://billduncan.org/implementing-service-level-objectives/) - I picked up a new book recently that is a great companion to the other SRE books published by O'Reilly. This book[1] goes much deeper into defining and refining SLOs than the chapters devoted to the subject in the original SRE books.[2][3] SLOs are not a panacea, but can be central to defining what level
- [NOTAM for SREs](https://billduncan.org/notam-for-sres/) - In aviation, NOTAMs are "Notices to Airmen" for conditions that are generally temporary and hence not information published in the usual places. NOTAMs[1] are used for alerting aircraft pilots of potential hazards along a flight route or at a location that could affect the safety of the flight. They normally contain information concerning conditions or
- [Traffic At 2 O'clock!](https://billduncan.org/traffic-at-2-oclock/) - Does your SRE team encourage help with on-call?
- [Own It !!](https://billduncan.org/own-it/) - How site ownership can reduce the stress of on-call through proactive maintenance. On-call is interrupt-driven. Everything else is "Owned".
- [SPOFs and Partial Panel](https://billduncan.org/spofs-and-partial-panel/) - While looking for SPOFs that you can eliminate, don't neglect your monitoring or you could end up flying blind!
- [Command Line Interface Guidelines](https://billduncan.org/command-line-interface-guidelines/) - A set of guidelines for creating user and tool friendly command line tools
- [Event Logs and A.I.](https://billduncan.org/event-logs-and-a-i/) - Using AI with event logs
- [Systems and Gardening](https://billduncan.org/similarities-between-systems-and-gardening/) - It's been awhile since I've written. I've been busy coming up to speed as an SRE with an awesome new team! It's gardening season up here in the Northern Hemisphere and while I was dealing with some trees and bushes that had died with a recent ice storm, I thought about the similarities in dealing
- [Riddler: Can you solve the not-so-corn maze?](https://billduncan.org/riddler-can-you-solve-the-not-so-corn-maze/) - I love AWK and I've written about it before
- [The Tail at Scale Approximation](https://billduncan.org/the-tail-at-scale-approximation/) - Quick and easy approximation for the probability formulae
- [The Tail at Scale Revisited](https://billduncan.org/the-tail-at-scale-revisited/) - Some charts for a better understanding of the relationship of backend objectives and how they impact user experience.
- [Hiring Questions, Problem 3](https://billduncan.org/hiring-questions-problem-3/) - This was an interesting question, so I thought I'd share it here.. Designing a rate limiting function.
- [Remote Work](https://billduncan.org/remote-work/) - Many companies and individuals are contemplating remote work now.
- [BPF Performance Tools](https://billduncan.org/bpf-performance-tools/) - BPF is one of the Swiss Army Knife tools for Performance Engineering on Linux.
- [Event Logs and K.I.S.S.](https://billduncan.org/event-logs-and-k-i-s-s/) - Looking for event logging solutions? Check this out!
- [Reading Week #5](https://billduncan.org/reading-week-5/) - Fun article on insanely large numbers and one on Realistic Goals.
- [Reading Week #4](https://billduncan.org/reading-week-4/) - Monitoring the SRE Golden Signals, an excellent overview by Steve Mushero
- [Reading Week #3](https://billduncan.org/reading-week-3/) - Some websites with weekly collections of relevant articles for your reading pleasure..
- [Reading Week #2](https://billduncan.org/reading-week-2/) - Investigating problems using log files
- [Works For Me #1](https://billduncan.org/works-for-me-1/) - Hacking thy self. Sharing some productivity tips that I use
- [Reading Week #1](https://billduncan.org/reading-week-1/) - Starting a new series of articles for interesting reads related to SRE and DevOps, or just technology and how it impacts us in general.
- [WordCamp US 2018](https://billduncan.org/wordcamp-us-2018/) - #WCUS happening this weekend. WordPress 5.0 and Gutenberg news.
- [Last Week's Security Earthquake](https://billduncan.org/last-weeks-security-earthquake/) - Last week there were two earthshaking security events. Yes, the Marriott data breach was big, but I'd like to talk about the one you might not have heard of.. About twenty years ago, ESR gave us The Cathedral and the Bazaar. One of the major themes in the book is that, "given enough eyeballs, all
- [Off Topic #1](https://billduncan.org/off-topic-1/) - The Reef's, Bermuda, photo from a recent vacation.
- [A Steal on O'Reilly DevOps/SRE Books!](https://billduncan.org/a-steal-on-oreilly-devops-sre-books/) - Limited time deal on 15 O'Reilly DevOps/SRE books for $15
- [Geolocating Your Users](https://billduncan.org/geolocating-your-users/) - I first became interested in geolocating a few decades ago while designing an email filtering system for some customers and noticing that most of the ``spam and malware'' originated in half a dozen countries.. History Back then, there was a website I'd found for identifying locations of IP indexes[1]. Fortunately, my customers were fine with
- [Hiring Questions, Problem 2](https://billduncan.org/hiring-questions-problem-2/) - A quick script for finding duplicate files.
- [(Ab)use of the R Language](https://billduncan.org/abuse-of-the-r-language/) - For years I've done most of my log scraping and analysis with the usual suspects; bash, sed, awk, perl even. The log scraping still uses those tools, but lately I've been toying around with "R" for the analysis. We're also going to demonstrate how inadequate it is to characterize system performance (or user experience) using
- [One Thing At A Time..](https://billduncan.org/one-thing-at-a-time/) - Learning things from any idea, test, change, upgrade in production
- [All Day DevOps 2018](https://billduncan.org/all-day-devops-2018/) - If you missed any part of the amazing All Day DevOps event, videos and resources are now available.
- [Seeking SRE](https://billduncan.org/seeking-sre/) - New O'Reilly book on SRE, building on the previous two successful books.
- [Realtime Component Request Deficit](https://billduncan.org/realtime-component-request-deficit/) - Component Saturation Factor, a reliable method for detecting performance issues and grey failures
- [awk, the Often Ignored Little Language](https://billduncan.org/awk-the-often-ignored-little-language/) - Many people use awk for one-liners; picking out fields from logs, doing pattern matching. It's capable of so much more however. IMO, the "littleness" of the language is one of it's strengths. Over a decade ago while commuting on a train to a customer site, I was amazed at the number of people working on
- [From the Get Go!](https://billduncan.org/from-the-get-go/) - Learning a new computer language can be fun! My first impressions of the golang..
- [Solving the Right Problems](https://billduncan.org/solving-the-right-problems/) - Ask the right questions, listen carefully and make sure that you're not hearing just what you want to hear..
- [Don't Panic!](https://billduncan.org/dont-panic/) - Some thoughts about handling critical system issues at scale.
- [Shades of Grey](https://billduncan.org/shades-of-grey/) - System failures don't have to be black and white. Catastrophic failures are easy to detect, partial failures, not as much.
- [Snowflakes](https://billduncan.org/snowflakes/) - Avoid snowflakes in the cloud or at scale
- [Averages Mostly Suck at Almost Everything..](https://billduncan.org/averages-mostly-suck-at-almost-everything/) - ..unless you're dealing with baseball. When dealing with systems, many of us think "Average" is a measure of "Typical" or "Normal". Many systems people will also use averages to look for "Abnormal". However, average (or mean) doesn't represent either "normal" or "abnormal" very well.. Looking for "Normal" Engineering and the business unit might like to
- [vi or emacs?  Really?!?](https://billduncan.org/vi-or-emacs-really/) - RTFM, (Read The Friendly Manual). vi or emacs editors, leverage your knowledge on the command lines! bash, less, mysql, python, R, tclsh et al.
- [Interactive bash Scripts](https://billduncan.org/interactive-bash-scripts/) - Building interactive commands that uses editing history and tab completion can be easy in bash
- [Must Have Books.. Another One!](https://billduncan.org/must-have-books-another-one/) - The Site Reliability Workbook
- [The DevOps Alternative](https://billduncan.org/the-devops-alternative/) - In a previous article, ``There's Always a Problem'', I described situations that can arise with the "Engineering vs. Operations" old way. The new way is a DevOps culture.. If you haven't read "The Phoenix Project"[1] or "Effective DevOps"[2], you need to. DevOps is more than just the tools, it's a whole new way of doing
- [CI/CD and Optimization](https://billduncan.org/ci-cd-and-optimization/) - When we talk of CI/CD we're often referring to Continuous Integration and Delivery while Optimization refers to Services/Systems. What I'd like to discuss is Constant Improvement/Continuous Development and Self-Optimization.. Being geeks, we are often looking at improving system/service delivery, performance and optimization. But what about ourselves? DevOps is more than just tools, it's a mindset.
- [Deep Dive, EPL Dotplots](https://billduncan.org/deep-dive-epl-dotplots/) - While working at RIM, I had the privilege of working with some brilliant engineers. During that time I developed a few of the techniques that I'll be describing; the EPD (Event-Pair-Difference) graph described in my previous post and the EPL (Event-Pair-Latency) Dotplot are a few of them. Background The example I've chosen to use
- [There's Always a Problem](https://billduncan.org/theres-always-a-problem/) - Do you have insatiable curiosity and are driven by a relentless pursuit of the truth? You might make a great problem solver, but be careful how you deal with your findings! I've been called a "bulldog" with problems; never letting go until I've found the root cause. I also have a curiousity that drives me
- [Preventing Trainwrecks](https://billduncan.org/preventing-trainwrecks/) - I keep a large framed photo of this on the wall in my office to remind me what can happen when things go "off the rails".. This was the Montparnasse derailment in 1895. You can read more about it here. Much of what I want to describe in this blog is about preventing situations like
- [Hiring Questions, Problem 1](https://billduncan.org/hiring-questions-problem-1/) - [latexpage] A colleague of mine once posted a hiring question to ask prospective developers: "What is the least significant 10 digits of the series: $1^{1}+2^{2}+3^{3}$ .. $1000^{1000}$ ?" The trick of course is to recognize that it's solved with modular exponentiation. He presented a page of java code which was akin to what he was
- [Look Up the Stack!](https://billduncan.org/look-up-the-stack/) - If you've been around systems long enough, you know that opportunity for performance gains goes up dramatically, the further up the stack you look.. Engineering will often suggest adding more hardware to increase performance. This will be somewhat less than a linear improvement however. To double the performance, double the hardware and hope that there
- [WSMeter: Performance Evaluation for Warehouse-Scale Computers](https://billduncan.org/wsmeter-performance-evaluation-for-warehouse-scale-computers/) - Many of us have dealt with making changes in production environments, possibly against hundreds or thousands of systems and, we'd like to know how the change impacted performance. It was with this in mind that I eagerly read through the paper describing WSMeter. You can read an excerpt here: WSMeter: A performance evaluation methodology for
- [Friday the 13th One Liner](https://billduncan.org/friday-the-13th-one-liner/) - Just for fun, how many combinations of months are there where Friday falls on the 13th? This one-liner will print out a table of month combinations along with the years for a given range. for y in {2000..2050};do for m in {1..12};do cal $m $y|awk 'FNR==1{m=$0}/^ 1/{print m}';done;done|awk '{a[$2]=a[$2]" "$1}END{for (i in a) print i,a[i]}'|awk
- [Don't Aggregate, Consolidate!](https://billduncan.org/dont-aggregate-consolidate/) - In previous posts, I've emphasized that averages are particularly bad at characterizing most things that you might be looking for. However, storing aggregated data of any type can limit your ability to analyze data later. If you're involved in photography, you know that storing original negatives or raw files gives you much more latitude in
- [Bitrot, Part 2](https://billduncan.org/bitrot-part-2/) - This article has a link to a simple script I've used for over a decade to detect corrupted files. It will detect and report on files that have changed, been added, deleted or possibly moved within the same directory structure. Obviously, this is mostly for "read-only" or "read-mostly" files that won't normally change. Detecting corruption
- [Bitrot, Part 1](https://billduncan.org/bitrot-part-1/) - Hidden data corruption in files

## Pages

- [Contact bill duncan](https://billduncan.org/contact/) - Let me know if you have any questions and/or feedback.. email: billduncan-blog (at) servermetrix (dot) com https://www.linkedin.com/in/servermetrix/
- [Interesting Links](https://billduncan.org/references/) - Some websites and references that I find are valuable.
- [About Me](https://billduncan.org/about/) - I've been looking at system performance issues for, well, a very long time.. Early in my career I was a C/assembler programmer, database developer etc., midstream switching to operations for awhile and, for the last decade or so, find myself on the fence. Somewhere in the middle; what we now call "Systems Engineering" or "Site

## Categories

- [Uncategorized](https://billduncan.org/category/uncategorized/)
- [Detecting Performance Issues](https://billduncan.org/category/detecting-performance-issues/)
- [System Performance](https://billduncan.org/category/system-performance/)
- [Diagnosing Issues](https://billduncan.org/category/diagnosing-issues/)
- [Code Bits](https://billduncan.org/category/code-bits/)
- [Just-for-Fun](https://billduncan.org/category/just-for-fun/)
- [Data Corruption](https://billduncan.org/category/data-corruption/)
- [R](https://billduncan.org/category/r/)
- [devops](https://billduncan.org/category/devops/)
- [RTFM](https://billduncan.org/category/rtfm/)
- [SRE](https://billduncan.org/category/sre/)
- [Problem Solving](https://billduncan.org/category/problem-solving/)
- [golang](https://billduncan.org/category/golang/)
- [Documentation](https://billduncan.org/category/documentation/)
- [Geolocating](https://billduncan.org/category/geolocating/)
- [OT](https://billduncan.org/category/ot/)
- [Security](https://billduncan.org/category/security/)
- [Reading Week](https://billduncan.org/category/reading-week/)
- [WFM](https://billduncan.org/category/wfm/)
- [Event logs](https://billduncan.org/category/event-logs/)
- [Monitoring](https://billduncan.org/category/monitoring/)
- [RemoteWork](https://billduncan.org/category/remotework/)
- [Aviation](https://billduncan.org/category/aviation/)
- [K8S](https://billduncan.org/category/k8s/)
- [AI](https://billduncan.org/category/ai/)

## Tags

- [Event-Pair-Difference](https://billduncan.org/tag/event-pair-difference/)
- [Graphing Performance](https://billduncan.org/tag/graphing-performance/)
- [Detecting performance problems](https://billduncan.org/tag/detecting-performance-problems/)
- [average](https://billduncan.org/tag/average/)
- [mean](https://billduncan.org/tag/mean/)
- [data corruption](https://billduncan.org/tag/data-corruption/)
- [bitrot](https://billduncan.org/tag/bitrot/)
- [filesystem](https://billduncan.org/tag/filesystem/)
- [sudoku](https://billduncan.org/tag/sudoku/)
- [awk](https://billduncan.org/tag/awk/)
- [recursion](https://billduncan.org/tag/recursion/)
- [observability](https://billduncan.org/tag/observability/)
- [monitoring](https://billduncan.org/tag/monitoring/)
- [performance](https://billduncan.org/tag/performance/)
- [bash](https://billduncan.org/tag/bash/)
- [scripting](https://billduncan.org/tag/scripting/)
- [latency](https://billduncan.org/tag/latency/)
- [R](https://billduncan.org/tag/r/)
- [distribution](https://billduncan.org/tag/distribution/)
- [graphing](https://billduncan.org/tag/graphing/)
- [ggplot2](https://billduncan.org/tag/ggplot2/)
- [system performance](https://billduncan.org/tag/system-performance/)
- [software stack](https://billduncan.org/tag/software-stack/)
- [diagnosing issues](https://billduncan.org/tag/diagnosing-issues/)
- [engineering](https://billduncan.org/tag/engineering/)
- [operations](https://billduncan.org/tag/operations/)
- [LaTeX](https://billduncan.org/tag/latex/)
- [modular exponentiation](https://billduncan.org/tag/modular-exponentiation/)
- [preventing disaster](https://billduncan.org/tag/preventing-disaster/)
- [SRE](https://billduncan.org/tag/sre/)
- [site reliability engineering](https://billduncan.org/tag/site-reliability-engineering/)
- [O'Reilly Books](https://billduncan.org/tag/oreilly-books/)
- [Checklists](https://billduncan.org/tag/checklists/)
- [Runbooks](https://billduncan.org/tag/runbooks/)
- [Aviation](https://billduncan.org/tag/aviation/)
- [log files](https://billduncan.org/tag/log-files/)
- [Reading Week](https://billduncan.org/tag/reading-week/)
- [on-call](https://billduncan.org/tag/on-call/)
- [ownership](https://billduncan.org/tag/ownership/)
- [environments](https://billduncan.org/tag/environments/)
- [SPOF](https://billduncan.org/tag/spof/)
- [Single Points of Failure](https://billduncan.org/tag/single-points-of-failure/)
- [dashboards](https://billduncan.org/tag/dashboards/)
- [partial-panel](https://billduncan.org/tag/partial-panel/)
- [time series data](https://billduncan.org/tag/time-series-data/)
- [event logs](https://billduncan.org/tag/event-logs/)
- [AI](https://billduncan.org/tag/ai/)
- [ML](https://billduncan.org/tag/ml/)
- [DevOps](https://billduncan.org/tag/devops/)
- [Remote Work](https://billduncan.org/tag/remote-work/)
- [WFH](https://billduncan.org/tag/wfh/)
- [#remotework](https://billduncan.org/tag/remotework/)
- [ratelimit](https://billduncan.org/tag/ratelimit/)
- [hiring](https://billduncan.org/tag/hiring/)
- [interview](https://billduncan.org/tag/interview/)
- [cli](https://billduncan.org/tag/cli/)
- [commandline](https://billduncan.org/tag/commandline/)
- [NOTAMS](https://billduncan.org/tag/notams/)
- [k8s](https://billduncan.org/tag/k8s/)
- [kubernetes](https://billduncan.org/tag/kubernetes/)
- [krew](https://billduncan.org/tag/krew/)
- [technical writing](https://billduncan.org/tag/technical-writing/)
- [tech documentation](https://billduncan.org/tag/tech-documentation/)
- [kubectl](https://billduncan.org/tag/kubectl/)
- [openssl](https://billduncan.org/tag/openssl/)