When does visibility become noise?
Every tool has some kind of activity, and that activity can create logs, notifications, and alerts. Every single one is a flow of data.
It is manageable when it is a trickle of a few alerts here are there, but once it grows to a flood of alerts, it can be a real challenge for one person to sort out.
The more you have to sort through these alerts every day, the more it desensitizes you to the content of those alerts. Small factors might indicate a serious issue but they become lost in the blur.
The human factor has limits like any system
Psychological factors like this are often overlooked when building operational security. We measure the capacity of our networks and systems, but we miss the capacity for a human to make meaningful decisions.
This may sound like a small thing, but it can make the difference between a quick response to a security incident.
A missed issue can multiply into a huge loss
This means that with more tools and greater visibility, the human element needs to considered.
Instead of latency, storage space, or encrpytion modes, we may need to look at mental health, culture and communication patterns. These often don't translate directly into simple quantitative measurements. They require the attention to understand the people who are responding to security.
Human capacity requires human solutions
Working better with humans isn't always an exact science, but there are a few ways you can ease the pain of alert fatigue:
- Ask what is most often taking up the attention of responders? Ask yourself if these are necessary? Can some versions of them be suppressed. Could they be sent as an aggregate report?
- Make sure that each alert has a meaningful triage classification. Define how much risk each alert could represent and separate them into different feeds. Make sure that reviewing the urgent alerts can be a straightforward a possible. As your systems grow, review these classifications to see if they still make sense.
- Move your alerting away from static detection rules and add behavioural alerts. Behavioural alerts work by measuring a baseline range of normal behaviour and then alerting on anomalous activity that falls out of that range. Behavioural alerts can reduce a lot of noise produced by normal routine activites.
- Check what else might be putting strain on security monitoring. The causes of fatigue aren't always obvious. Review how responsibiltiy is divided, and how communication is handled. Check for areas of ambiguity that might lead to stress of confusion on the team.
People have different requirements than technical systems, and have capabilities that computers can't hope to replicate. Make sure you are getting the best from the people on the front lines.