When it comes to cybersecurity, you have to be right every time; malicious actors only have to be right once. To better understand your organisation’s cyber risk and the evolving threat landscape, you must be intelligence-led, with cybersecurity metrics high on your agenda. Read on as we delve into the golden rules of measuring what matters and how strategic intelligence informs better security decisions.

Six reasons to take advantage of metrics

One: To learn

To find out something about our environment that we didn’t know before, we have to ask ourselves, what questions are we trying to answer with the information gleaned? Take strategic intelligence; it helps decision-makers better understand their organisation’s specific cybersecurity risks.

Two: To decide

We want the results and value of the measurement to inform and help us make management decisions.

Three: To implement or change a course of action

Usually, the decisions we make result in a course of action, it might change the size of an existing action or result in a cessation of activity, but there is a real need to change what we are doing.

Four: To validate hypotheses

We have questions that we want to answer. Ideas we want to test and suppositions that need backing up or disproving.

Five: To build confidence

If we’ve previously acted, it might be that we want to see how that action has panned out. It validates the actions and gives the opportunity to course-correct. Done right, it allows course correction to be less and less like a guess over time, subject to other parameters staying relatively stable.

Six: To tell a story

Within a pack of information seeded with metrics, the reader can go on a journey to explore and to experience a story. What that story is, depends on the data, both in type and in-depth.

Image for: Secrutiny - Cybersecurity Metrics: Are You Measuring What Matters?

Figure 2: Cybersecurity Metrics Illustration

What makes a good and bad metric?

A good metric is explicable, understandable and allows comparison internally (looking at the metric over time to get a useful steer or opportunity for course correction) and externally (benchmarking against other organisations in the same sector of a similar size, subject to the information being available).

When delving into metrics, be careful not to waste time measuring things you have no control over or those that never change – ensure you can still collect the metric if it will be used ongoing in reporting. A good place to look for pointless metrics is in vendor dashboards, as development teams on a time constraint often choose these.

Examples of good metrics include:

  • CIS benchmark score per host – this allows you to see if you’ve got accidents that occur across your standard build images.
  • Vulnerabilities per host – this takes your whole populous into account and is relatively stable with a changing threat landscape so you can make the material differences and improvements.
  • Kill chain segment time to execute – i.e., move laterally successfully.
  • Percentage of AV/EPP events successfully handled.
  • Cost per control event – for a given control divide the cost by the effectiveness of the events.
  • Previously fixed vulnerabilities that have returned.
  • Malware instances that have successfully executed.

ISO 27001 encourages us to set the balance between investment in security risk reduction and the harm likely from security-related failures – this balance can help choose what to measure.

Bad metrics include:

  • AV detection – this doesn’t explain whether there were successful quarantines/deletions, how well signature updates are performing while people are WFH or the mechanism by which they arrived. So, trying to take action on it will likely be futile.
  • Number of security incidents raised – this is similar to the AV metric. Does it mean that your ability to detect incidents has improved, the number of available incident types has grown with the threat landscape, or if a significant change in infrastructure has prompted a weakness in another security control?
  • Number of closed vulnerabilities – not all vulnerabilities are closed with a simple fix or a patch. Many cases represent more a “class” of issue that needs addressing.
  • Security tickets closed – this metric doesn’t paint the picture of how many tickets there were to start with or if the number closed is supposed to be positive or negative.
  • The log management metric – this is simply a vanity metric.

So, metrics are incredibly beneficial in helping organisations demonstrate how well it’s achieving its cybersecurity risk reduction goals. Still, when embracing the added value of strategic intelligence, this can be further improved, providing invaluable context and focus for your cyber defences.

What You See Is All There Is (WYSIATI)

When you look at strategic intelligence, there is a danger that ‘What You See Is All There Is (WYSIATI)’, a theory by psychologist and economist Daniel Kahneman. It states that when the mind makes decisions it deals primarily with ‘Known Knowns’, phenomena it has observed already. It rarely considers ‘Known Unknowns’, phenomena that it knows to be relevant but about which it does not have information and appears oblivious to the possibility of ‘Unknown Unknowns’, unknown phenomena of unknown relevance. He explains that humans fail to consider the complexity and that their understanding of the world consists of a small, unrepresentative set of observations.

What does that mean for your cybersecurity strategy? The primary thing to remember is don’t rely on your memories. Instead, use imagination to conceive of ways to avoid future risk by taking different paths and options. In technology and security, it’s possible to implement a new solution that dispenses with dealing with a security risk yet provides significant business benefit.

For example, if you’re looking at how to protect an internet-facing website that is running direct or through a proxy and must fend for itself, trying to reduce the risk directly will be time-consuming, costly and marginally successful. If money is invested in a CDN with DDoS and WAF capability, you’ll see an uplift in security but also in availability and speed to load. Whilst leapfrogging several epochs of security technology to reach the current state of the art can offer huge dividends.

So, as far as metrics are concerned, you need to see what information you can collect and attempt to use things that exist in the control environment. And utilise strategic intelligence to add analysis to information and make course corrections.

But, remember gathering metrics only for metric sake does not survive technology change very well. You must plan to collect information for the long haul for the benefits of trending to be realised. So, don’t wait until you have the perfect set of measurements; start collecting and using the information they offer to refine. It’s about challenging yourself to check if your board’s questions will be answered or supported by the information you’re collecting.

If you would like further information on this topic or speak to one of our experts, please click here.