Don’t Panic!

Some thoughts about handling critical system issues at scale..

As far as availability goes, we don’t have much control over MTBF. Things break. We have more control MTTR, which really includes the time to detect and respond to failures as well.

There are several goals in any incident response..

  1. Primary goal is the customer experience and trumps all others.
  2. Communicate to the customers if service is or was impacted.
  3. What can we learn about the incident to make it unique and interesting?
  4. How can we improve our response to incidents? Hindsight is always humbling. I can’t think of any incident I’ve handled where I didn’t think, “why didn’t we see this sooner?“.  (Hopefully because they weren’t as memorable and we handled it more quickly..)
  5. ..and of course, fix it!

If you are involved in incident response, here are some pointers that I’ve learned over the years.

  1. Don’t panic! We think more logically when we remain calm. Encourage focus.
  2. As I’ve pointed out here, the first priority is to keep the airplane flying (or the service servicing). Find a way to mitigate the problem and save the customer experience as much as possible.
  3. Draw boundaries around the issue; timing, scope and severity.
  4. Communicate and call on the expertise required.
  5. Encourage fact finding to develop the theories and anti-theories with everyone.
  6. Use whatever methods available for recording what transpires, including things that look ok and discarded theories.
  7. Try not to trample on any evidence that might be used later to dig further.

Some (not exhaustive) techniques I’ve used in drilling down to fix issues:

  1. What changed? Recent rollout history? Nothing happens randomly except stray gamma rays..
  2. Bisection, What is working well, as well as what’s not.. Find the boundaries.
  3. Any similar issues in the past? Look up old RCA’s, tickets..
  4. In fact, recent past tickets may have clues.
  5. Artificial ignorance; look for log lines or other events that we don’t normally see. Basically, filter out “normal” and what you should be left with is abnormal.
  6. Communicate and encourage people to keep looking and reporting what they see, including normal stuff.
  7. Look for simple explanations.
  8. Narrow down the timing of the incident start, as this may give clues to cause.
  9. Develop some theories and counter theories.
  10. Process of elimination, rule some theories out.
  11. Test hypothesis if practical and won’t interfere with customer experience.
  12. Top down approaches sometimes works, drill down from the applications.
  13. Assume nothing, follow the data..

The approaches you use may vary depending on the tools, instrumentation and the specific circumstances.

What works for you?