The Network is Crying Wolf

Real Numbers Support Better Detection Accuracy

Data shows that automated response is not the big objective for security operations. While startups and investors like the automated response pitch, reviewing actual operations data shows that the real need is for accurate detection. Data shows that the number of critical alerts over the last year has consistently increased, but the number of incidents has decreased. This means that there are more issues to validate, but less real issues to address. The greatest efficiency gain is in reducing the number of false alerts. On the other hand, automating without validation is likely creating unnecessary prevention and denial of service.

Fluency is blessed with good customers who talk about their needs. I have to admit that I like the orchestration pitch and was debating making Fluency a pure orchestration tool. That is a big decision, one that requires talking to your customers. I sent out emails, picked up the phone, and met face to face. I wanted to know where the operation’s effort truly was.

I focused my questions on the number of incidents that needed to be addressed. I often see the use of vanity metrics. The most significant vanity metric for a security operations center (SOC) is to talk about the number of alerts. I was at the Gartner Summit hearing two large companies talking about millions of alerts a day, but not a single measurement of their response capability. The number of alerts does not translate into success, but merely available information for analysis. The objective of a SOC is its response.

Talking to our customers, there was a consistent answer in their data. The number of incidents per month was decreasing, while the number of critical events being reported was increasing. More than one customer showed a consistent decrease in their validated incidents, while their critical alerts more than doubled in the last year. This particular ratio showed an increase of critical alerts-to-incidents from 20-to-1 to 42-to-1.

Fluency is unique in that it does real-time analytics for detection and validation as part of its orchestration. This is a critical aspect as we feel that automating a response to an incorrect alert can be as damaging as missing a real one. The false positive ratio of critical alerts tells us that you are forty times more likely to respond to an incorrect alert if you do not validate it.

What does this ratio of critical alerts to confirmed issues mean? It means that security products are crying wolf more often. It means that security products are alerting more often and are increasingly incorrect in their alerts. The fact is that there are more alerts saying they are critical, and this increase is putting a real strain on staff to review the alerts. It also means that companies that focus on responding without validating alerts will be doing more harm than good.

As for Fluency, we will continue to perform orchestration but with added emphasis on analytics and validation. This is where our customers need us. Fluency is one of a couple companies that perform real time analytics using machine learning. Most machine learning approaches are with static searches or are just statistical analysis rebranded as machine learning.

Chris Jordan is CEO of College Park, Maryland-based Fluency (, a pioneer in Security Automation and Orchestration.