Every post-incident review that includes the phrase "the alert did fire" is really a review about alert design. The detection worked. A human being looked at it, or could have, and nothing happened. That is not a failure of attention. It is a failure of the system that demanded the attention.
Why people stop reading
Consider what a security channel looks like in a typical mid-sized organization. Several hundred notifications a day. The overwhelming majority are benign: a developer logging in from a coffee shop, a scheduled job whose permissions look unusual out of context, a scanner probing an endpoint that is properly locked down anyway.
A person watching that channel learns, correctly, that the base rate of real problems is near zero. Ignoring it becomes the rational response, because investigating everything would consume their entire week and produce nothing. By the time a genuinely serious alert arrives, the channel has trained everyone to scroll past it.
Telling that team to be more vigilant does not work, and it is the wrong target. They responded sensibly to the incentives the system created.
The arithmetic nobody runs
Suppose a detection rule is 99% accurate — which would be a very good rule — and the thing it detects occurs genuinely about once a month. Run it against a million events a day and the false positives arrive by the thousand. The one real event is somewhere in there, indistinguishable at a glance from the rest.
This is base-rate arithmetic, and it is unforgiving. When the underlying event is rare, even excellent accuracy produces overwhelming noise. Adding more rules makes it worse, not better, which is the opposite of the instinct most teams have after an incident.
Adding a detection rule after every incident feels responsible. It is how channels become unreadable, one reasonable decision at a time.
What actually helps
Correlate before notifying
A single unusual login is noise. An unusual login, followed by enumeration of storage buckets, followed by an access key being created, is a story. Alert on the story. Almost every alert worth waking someone for is a sequence rather than a single event, and most tooling can be configured to wait for corroboration before it escalates.
Attach the context the responder will need
An alert that says "unusual API call from principal X" forces the responder to go and find out what principal X normally does, who owns it, and what it can reach. Most of that lookup can be automated and attached to the alert itself.
The difference between a thirty-second triage and a thirty-minute one is usually just whether the context arrived with the notification.
Give every alert an owner and an action
If nobody can say what a responder should do when a particular alert fires, it should not be firing to a human. Route it to a dashboard for periodic review, or delete the rule. An alert with no defined response is a notification that trains people to ignore notifications.
Separate the channels
Wake-someone-up alerts and look-at-this-tomorrow alerts belong in different places. Mixing them guarantees that the urgent ones inherit the response time of the routine ones, because people cannot tell them apart at a glance.
A useful exercise Take last month's alerts. For each rule, count how many fired and how many led to an action. Any rule with a large numerator and a zero denominator is actively harming you — it is spending attention and returning nothing.
The uncomfortable part
Tuning alerts properly means accepting that you will detect slightly less. A rule tightened to reduce false positives will occasionally miss something it would previously have caught.
Teams resist this, because a missed detection is legible and attributable in a way that slow erosion of attention is not. But the comparison is not between tuned alerting and perfect vigilance. It is between tuned alerting and a channel nobody reads — and a rule firing into a channel nobody reads has a detection rate of zero regardless of its theoretical accuracy.
What good looks like
A security channel that a person can read in full, every day, without dread. Alerts arrive with enough context to triage in under a minute. Each one has an owner and a defined next step. When something fires at 3 AM, the responder believes it is worth getting up for — because experience has taught them it usually is.
That is achievable in most environments. It just requires treating alerting as a product with users, rather than as a coverage checklist to be maximised.
Getting there is a large part of what our managed detection and response work involves — the monitoring matters, but the tuning is what makes anyone trust it.