Field note
Reading failover drills as reliability evidence
Failover exercises produce some of the richest material for cluster reliability reporting — if you capture more than a pass/fail tick.
A green drill status hides almost as much as a green CPU chart. The useful material is the timeline: which health checks lied, which consumers lagged, which humans improvised a step that is not in the runbook.
When Cluster Pulse Point includes drill notes in a reliability report, we ask for the raw chat export or bridge recording excerpt, not the summary slide. Application analytics from the drill window often show saturation patterns that never appear in weekday traffic.
What to keep from the next drill
- Exact clock times for declare, failover start, and customer-visible recovery
- List of manual steps that were not automated
- Any “we always do this” habit that exists only in one engineer’s head
Those fragments turn a ceremonial exercise into evidence. Without them, cluster reliability reporting has to lean on production incidents alone — and production is a harsh editor.