(Ab)use of the R Language, Part 2: latencylens

Most graphs we see as SREs are time based along the X axis, whether it is request rates, error rates, latencies or some other parameter along the Yaxis.

What I would like to show you is an alternative which is useful sometimes in dealing with latencies and first mentioned in (Ab)use of the R Language.
This technique trades the granularity of time for better visibility of the “shape of latencies” and how it impacts “customer experience”.

It is often useful for looking at how the shape has changed between two different time ranges. Useful when you want to compare:

• engineering who want to see how their changes have impacted latencies across a release
• customer service who want to see what has changed for a customer when speaking with them, perhaps looking at SLA’s if there are agreements in place
• incident commanders who are looking at what (or who) was impacted in an incident.

As I mentioned in the original article, kernel density graphs are like histograms, but enable you to compare two different time ranges, regardless of the difference in the number of events. For example, you can compare today’s data with the previous week, or month. You can slice and dice it by service, route or customer to let engineering know how the application has improved (or not) or customer service can relate to customers about what they are seeing.

Why look at the shape at all

An average collapses a distribution to one point. A p95 collapses it to one point too, just a different one. Neither tells you the thing that actually matters for debugging or for talking to a customer: how many different experiences are bundled into that number.

Most services don’t have latencies centered around a single value, they have several peaks or modes depending on context or what’s going on in the backend. A cache hit and a cache miss. A warm connection and a cold one. A request that stays in-region and one that has to hop across. Each of those shows up as its own mode — its own peak in the kernel density graphs. Look at a given service’s curve enough times and you start to recognize its normal shape: how many peaks it has, roughly where they sit, how tall each one is relative to the others. It becomes a kind of fingerprint for that service.

That’s useful in two different ways for two different audiences:

• A curious engineer watching that fingerprint change has something
  concrete to chase. A new peak appearing, an existing one shifting
  right, a valley between two peaks filling in — none of that shows up
  in an average, but all of it shows up immediately in the shape, and it
  points at a specific latency range to go drill into instead of a
  vague “latency is up 8%.”
• A customer service rep looking at one customer’s curve sees which
  peak that customer is actually landing in, not just where their
  average sits. “Average latency is 400ms” could mean every request is
  a boring, consistent 400ms — or it could mean 90% of requests are a
  snappy 50ms and the rest are a miserable 4 seconds, averaging out to
  something that sounds fine on a dashboard and doesn’t feel fine to the
  customer describing it on a call. The two peaks are the actual story.

What’s better about the graph

I’ve improved the script over the years and generalized it enough for wider use. The core idea — overlay two periods as filled kernel density curves on a log x-axis — was already in the original piece. I’m also making it available for others to use in my github repo.

Some of the recent improvements:

• Curves are filled, not just outlined.
• The stats block moved out of the way to the right.
• Log-x with explicit breaks, sqrt-y.
• Jittering the low end. Below 10 which smooths it out.

Here is an example using data similar to what the original article used. You can compare the results with the original kernel density graph, which has a description of how to interpret it.

What’s open source, and what isn’t

The R script and a thin bash wrapper are in my GitHub repo.

That’s latencylens.r (the plotting logic above) and latencylens (a wrapper that validates arguments and calls Rscript). It doesn’t care where the data came from, as long as it’s handed a file in the shape described below. The search.dat file (used for the graph above) is bundled as a worked example. You can recreate the graph using a command like this:

./latencylens -a 0.5 -w 10 -s 30 -t "Sample Search" search.dat search

What’s not in the repo are the scripts that pull the data from your event logs and process them into the data files used. I generally use a combination of bash, awk and jq.

There’s also a Julia port sitting in the same project, using CairoMakie and KernelDensity.jl to produce the same figure. It’s not part of this release; if it holds up under more use I’ll write about that separately.

The data file, if you want to build your own

There’s no way to hand out one script that builds this file for everyone — it depends entirely on where your latency numbers live (load-balancer logs, an APM tool, a database of request records, whatever). But the file itself is deliberately plain, so here’s what it needs to look like if you want to feed it your own data.

It’s a whitespace-delimited text file with a header line and exactly three columns:

time value latency

• time — which of the two periods this row belongs to. The file
  must have exactly two distinct values in this column, no more, no
  fewer — that’s what tells the tool which two things it’s comparing
  and what to put in the legend. The two values can be almost anything:
  two dates, or plain labels like
  baseline/incident, before/after, control/canary. The tool
  doesn’t interpret them, it just needs there to be two.
• value — whatever you want one panel per: a customer ID, a
  route or service name, a region, a canary-vs-control cohort tag.
  Every distinct value in this column gets its own panel.
• latency — a plain number, in milliseconds, microseconds or seconds. Fractions are fine. They just need to be the same units.

One row per request. Both periods go in the same file, distinguished only by the time column — so building your own version is usually: pull the records for period one, emit <time-label> <group> <latency> per row; do the same for period two with the other time label; concatenate both into one file under the header line. awk and jq over structured logs get there easily.

Same conclusion as last time

When comparing latencies across two time ranges to see what has changed, averages or other single numbers hide more than they show.
The kernel density distribution graphs give you a better picture of how the systems are performing or show you what your users are actually experiencing.