Our on-call engineer muted the pager after 200 false alarms that month. Alarm 201 was the real outage.
An alert should mean exactly one thing. A human needs to act right now. Everything else belongs in a dashboard or digest.
observabilityops
Problem
Our alerting fired on anything that looked abnormal: a CPU blip, a queue depth spike that cleared itself, a deploy restarting pods. None of it needed a human. All of it paged one.
By the end of the month, the on-call engineer had muted the channel. The alert for the actual outage (a real one, customer-facing) landed in the same muted channel as the 200 before it. It sat unread for 40 minutes.
Why it happens
Alerts get added the moment something goes wrong once, as a reflex, not as a decision. Nobody removes them later because removing an alert feels like removing a safety net, even when the net has never caught anything real. The alert list only grows, and every page trains the on-call engineer a little more to stop trusting the pager.
Better approach
An alert should mean exactly one thing: a human needs to act right now. Anything else (trends, warnings, “worth a look Monday”) belongs in a dashboard or a digest, not a page. Review the alert list on a schedule and delete anything that hasn’t led to action in the last quarter. If an alert fires and the fix is “nothing, it resolved itself,” that alert is the bug.
Example
A queue-depth alert that paged whenever depth crossed a static number, even during expected traffic spikes. It fired 40+ times a month and never once needed intervention. It sat next to the alert for a database running out of connections (the one that mattered) until someone finally looked at both in the same muted channel.