In shortThe timeline / The cause / The fix / The lesson.
1. The timeline
At 03:02, a runaway job started sending alerts every second. By 03:10 the on-call hen had silenced it and begun triaging.
2. The cause
A cron expression with a typo ran every second instead of every hour. It was reviewed, approved, and missed.
3. The fix
We added a dry-run that shows the next ten run times for every schedule. The typo would have been obvious.
4. The lesson
Review by simulation, not by reading. Humans are bad at parsing cron, and chickens are no better.
Written by Beatrice Featherstone, Head of Site Reliability. Disagree? Tell us. We read every message, usually before the second coffee.