Writing an Incident Response Runbook

Prerequisites: no specific tooling required — this project is written-artifact and process-focused. Familiarity with your own team's cloud environment (or the AWS environment from earlier projects in this roadmap) makes the exercise more concrete.

A runbook that only exists as tribal knowledge in one senior engineer's head is not a runbook — it's a single point of failure that happens to be a person, and that person is guaranteed to be unavailable at 2am during the actual incident. A real runbook is written down, specific enough that someone on-call who has never handled this exact incident before can follow it correctly under pressure, and — critically — has actually been tested against a simulated version of the scenario it describes.

You'll write a runbook for one of the single most common real incidents in any cloud environment: a credential (an AWS access key, an API token, a database password) has been accidentally committed to a public repository or otherwise leaked. Then, instead of treating the document as done once it's written, you'll run a tabletop exercise against it — walk through the incident out loud, step by step, exactly as written — and revise it based on what falls apart in the walkthrough. That revision step is the part almost every team skips, and it's the part that actually determines whether the runbook works.