Operational Readiness

Operational readiness is the state in which a service, its people and its tools are prepared to run it in production, and it must be proven, not declared. It rests on four assets. Alerts tell the team when to act, and must be symptom-based, actionable and owned. Dashboards show what is happening and answer the question 'is it us?' within 30 seconds. Runbooks tell a non-author what to do at 3 a.m. Ownership and on-call define who acts and who helps. The proof is a game day: if an engineer who did not build the service can resolve a simulated failure using these assets, it is ready.

A good runbook is written from the alert's point of view, for someone who did not build the service. It starts with the alert or symptom name and states the business impact, such as customers being unable to order at about ₹45,000 a minute. It links the exact dashboards and queries, already filtered. It lists safe actions first, such as aborting the latest rollout or disabling optional features, then decision points, such as going to section 4 if pricing latency exceeds 500 ms. It defines escalation by role or rota rather than by a person's name, and ends with verification criteria, such as error rate below 0.5% for ten minutes.

It should first show whether users are affected: SLO attainment and remaining error budget for the service's key journeys. Next come the golden signals per endpoint, meaning traffic, errors, latency percentiles and saturation, so on-call can see what is failing and since when. Below that, a dependency row shows the error rate and latency of each downstream call, answering whether the cause is us or them, followed by resources such as CPU, memory, pools and pods. Deploy, feature-flag and incident annotations overlay every graph to show what changed. ShopNow's redesign from 64 panels to this layout cut the median time to answer 'is it us?' from six minutes to under one.

A good alert means a human must act now, and it helps them do so. It is symptom-based, firing on what users experience, such as fast SLO burn or a failing synthetic checkout, rather than on causes such as high CPU, which belong on dashboards. It is actionable, with a clear next step, and owned, routed to the team that runs the service through catalog metadata. It links its runbook and dashboard in its annotations. It is tested, having fired in staging or a game day, and rare, because frequent pages create fatigue and get ignored. Slow-burn conditions should open tickets rather than wake people.

Manage them as code in the service repository, so they change in the same pull request as the code that affects them. Alerts are PrometheusRule YAML, dashboards are Jsonnet, Grafonnet or Terraform, runbooks are Markdown rendered in the developer portal, and ownership lives in a Backstage catalog-info.yaml file. Generate the baseline from platform templates so every new service starts complete. Then add feedback loops: every incident review checks whether the alert, dashboard and runbook helped and creates fixes; game days reveal stale steps; and readiness reviews check coverage metrics, such as the share of alerts with runbook links and of runbooks exercised recently.

A game day is a planned exercise in which the team deliberately injects a realistic failure to test whether detection and response actually work. You choose a scenario, such as a pricing latency spike, the loss of a Kafka broker or a bad deploy, and inject it in staging, or in production with guardrails, using AWS Fault Injection Service, a feature flag or a fault proxy. Then you observe whether the right alert fired and reached the right person, and whether the on-call engineer could resolve it using only the runbook and dashboard. Every gap becomes a ticket, and the exercise is repeated until resolution is fast.

Route by ownership metadata, never by hard-coded names or a shared channel. Each service's catalog entry in Backstage records its owning team, tier and PagerDuty service. Alert rules carry team and severity labels, and Alertmanager or the paging tool routes on those labels to the owner's escalation policy, with severity deciding between a page and a ticket. When ownership changes, one catalog edit updates routing everywhere. A CI check that fails when a service has no owner or an alert has no team label keeps the data complete. At ShopNow, routing by catalog owner cut misrouted pages from 14% to under 1%.

Runbooks start as documents and gradually become tools. When a step is repeated across incidents and is safe, such as collecting a thread dump and heap histogram, aborting a rollout or disabling an optional feature flag, it gets automated: a portal button that runs an SSM Automation document or Kubernetes job, or a ChatOps command such as /rollback checkout. Automation removes typing errors under pressure, records what was done in the incident timeline and cuts time to mitigation. Judgement steps stay human, but the runbook calls the tools. Every incident and game day feeds the runbook, adding missing steps and removing stale ones.

Prove it by exercise, with measurements, rather than by checking that documents exist. Run game days in which an engineer who did not build the service handles injected failures using only the alerts, dashboards and runbooks, and measure time to detect, time to reach the right person and time to resolve. Back this with coverage evidence: alert rules in the repository with test firing, a runbook link on every alert, runbooks exercised in the last six months, and the on-call schedule and escalation configured. A 30-minute readiness review checks the evidence before launch. ShopNow's pricing game day initially exposed a missing step and, after fixes, resolved in seven minutes.

Keep pages few and meaningful, the rota large enough, and the load visible. Page only on symptom-based, actionable alerts, and route slow burns and cause-based signals to tickets and dashboards; aim for fewer than two pages a shift and under one night page a week per team. Keep at least six engineers in a rota, and use follow-the-sun across time zones for Tier 0 services so nobody is routinely woken. Track pages and out-of-hours work per person, review noisy alerts at every retro, and give recovery time after bad nights. Exhausted on-call engineers make slow, risky decisions, so sustainability is itself a reliability requirement.