I look after a platform of 40-plus production databases in healthcare. Multi-terabyte, HIPAA-scoped, 99.9%+ availability targets, and a rotation of engineers who carry the pager for it. When I joined, the team was competent and the systems were reasonably instrumented. We still had a repeat-incident problem.
Not a volume problem. A repeat problem. The same class of page kept coming back, sometimes to a different engineer, often a week or two apart, and each time it got handled fresh. Somebody would work it at 3 AM, apply the mitigation that made the symptom stop, close the incident, and go back to sleep. Entirely reasonable behavior. Also the exact mechanism by which an organization pays for the same outage four times.
The instinct in that situation is to reach for tooling. Better alert correlation, a new observability tier, an incident platform with nicer timelines. We had Datadog and Grafana and they were fine. Tooling was not the constraint.
The constraint was that context died at the handoff.
What a readiness review actually is
The intervention was unglamorous: a short structured conversation before every rotation change, between the engineer handing off and the engineer picking up. Not a status meeting. Not a standup. A handoff with a fixed agenda, because an unstructured handoff degrades into “quiet week, good luck” within about three weeks.
Three things get walked, every time:
What paged, and why. Not the alert name. The actual causal story, or an honest “I don’t know.” An engineer saying “this fired four times and I never found the root cause” is the single highest-value sentence in the review, and the culture has to make it cheap to say. If admitting an unexplained repeat costs someone status, they’ll stop saying it and you’ll lose the signal entirely.
What changed in the platform. Deployments that landed, indexes added or dropped, a failover that happened, a maintenance window, an AG topology change, a new ingestion source that quietly tripled write volume on a table nobody was watching. The incoming engineer inherits a different system than the one described in the runbook from six months ago.
Which runbooks are stale. Every incident worked during the rotation either confirmed a runbook, corrected one, or revealed that one didn’t exist. That gets captured as work, not as folklore.
The whole thing fits in half an hour. The output is not meeting notes. The output is tickets.
Runbooks as code, not as wiki pages
The readiness review only compounds if what comes out of it lands somewhere durable. Wiki pages rot silently, because nothing about a wiki page tells you it’s lying to you.
We keep response playbooks in version control alongside the automation that supports them, which means a stale runbook shows up the same way stale code does: in review, with an author and a date and a diff. A trimmed example of the shape:
id: DB-OPS-014
title: Blocking storm during peak ingestion window
severity_hint: Sev-2 (Sev-1 if checkout path affected)
last_validated: 2026-07-14
validated_by: on-call readiness review
detect:
- signal: avg wait time on LCK_M_* exceeds baseline 5x for 3+ min
source: datadog monitor db.locks.wait_ms
- signal: blocked session count > 25 sustained
source: grafana panel "Blocking - prod fleet"
triage:
- step: Identify head blocker and its full input buffer
query: sql/blocking-head-of-chain.sql
- step: Confirm whether plan for the blocked statement regressed
note: compare against captured baseline plan before killing anything
mitigate_5min:
- Kill the head blocker ONLY after capturing session, plan, and input buffer
- If regression confirmed, apply the pinned plan guide (see durable_fix)
durable_fix:
- owner: platform-db
action: index or refactor per RCA; do not close incident on mitigation alone
- gate: repeat-incident review at next rotation handoff
known_repeat: true
last_repeat: 2026-06-02
Two fields there do most of the work. last_validated makes staleness visible. known_repeat and last_repeat make it structurally impossible to treat the fourth occurrence as if it were the first.
That durable_fix gate matters more than it looks. The default failure mode of a good on-call engineer is being too effective at mitigation. If the symptom reliably goes away in four minutes, the durable fix never gets prioritized, because it never hurts enough during business hours to make anyone’s roadmap.
Finding your repeats
Before you can review repeats you have to see them, and most alerting stacks are optimized for the present tense. A crude grouping over incident history gets you most of the way. Even something this blunt is usually enough to start the conversation:
SELECT
alert_key,
COUNT(*) AS fire_count,
COUNT(DISTINCT responder) AS distinct_responders,
MIN(opened_at) AS first_seen,
MAX(opened_at) AS last_seen,
AVG(DATEDIFF(MINUTE, opened_at, resolved_at)) AS avg_minutes_to_resolve
FROM ops.incident_history
WHERE opened_at >= DATEADD(DAY, -90, SYSUTCDATETIME())
GROUP BY alert_key
HAVING COUNT(*) >= 3
ORDER BY fire_count DESC, distinct_responders DESC;
Sort by distinct_responders and you find something specific and uncomfortable: the alerts where three or four different people each independently solved the same problem and none of them wrote it down. That column is a direct measure of institutional forgetting.
What it moved
Over the period we ran this, alongside better alert routing and standardized response playbooks, repeat incidents came down roughly 35% and MTTR roughly 30%. I want to be careful about the causal claim. Those two changes traveled together, and I can’t cleanly separate the readiness review from the alerting work. What I can say is that the alerting improvements were mostly identified in readiness reviews. The review was the mechanism that surfaced which alerts were lying, which thresholds were noise, and which pages had no runbook behind them.
The secondary effect was the one I didn’t predict. Newer engineers stopped dreading the rotation. Not because the systems got dramatically quieter at first, but because picking up the pager stopped feeling like inheriting an undocumented system from someone who’d already logged off. Toil and fear are closely related, and both are retention problems before they’re reliability problems.
The part I’m still working on
I don’t have this fully solved for cross-team dependencies. Our readiness review is excellent at transferring context inside the database platform and mediocre at capturing the application-side changes that end up landing on us as a blocking storm three days later. Right now that context arrives through relationships rather than through process, which means it’s real but not durable.
I’m experimenting with pulling deployment metadata from CI into the review automatically, so the “what changed” section starts pre-populated rather than depending on whoever happens to remember. Early, and I’m not sure yet whether it’ll add signal or just noise.
If you’re running on-call for a data platform in a regulated environment: how are you transferring context across team boundaries, not just across rotations? That’s the seam where I still see incidents get re-solved from scratch, and I’d genuinely like to hear what’s worked.
