Measuring the Effectiveness of Threat Feeds

We do a lot of important and sometimes complicated things as we try to defend organizations from cyber attacks. One thing that often gets left behind, or at least isn’t done as effectively as it could be, is measuring what we do and how well we do it. I think I’m speaking for more than just myself when I say that. I know this area has been a challenge for me personally over my entire career in security. And trying to measure effectiveness in the abstract world of threat intelligence makes the challenge that much more daunting. It’s not that it can’t be done; it’s more a question of not devoting enough time to metrics when there are so many other fires that need to be put out.So we go along without meaningful metrics or maybe we cobble together some spreadsheets for management, but few of us are very good at this stuff. Major kudos to those of you who happen to do this well by the way. There are things we honestly should be measuring and collecting beyond “here are how many blocked attacks we saw last month from our IPS” (or firewall, or antivirus, etc.). Knowing how well your processes and tools are performing is key to knowing where problems are or where to focus energy for improvement.Anomali Match has some really useful reporting built-in that can help gain some valuable insights in this area. Specifically, the Feed Contribution & Impact report. Running this report is easy - just a few steps - and it can be scheduled to run automatically at certain intervals. We recommend running this report once a week.To generate the report:<ol><li>From the main dashboard in Anomali Match, click Manage on the top menu and select Reports.</li></ol><img alt="" src="https://cdn.filestackcontent.com/ctgRp0tTSnalHy5OBsLM"/>Anomali Match: Generating a Feed & Contribution Impact Report Step 1 2. Once on the Reports screen, click the Actions menu and select Schedule Report.<img alt="" src="https://cdn.filestackcontent.com/Btt88SnyQd6JGiXXWl6z"/>Anomali Match: Generating a Feed & Contribution Impact Report Step 2 3. Fill out the form and be sure to select the Anomali Threat Feed Report for the Dashboard field.<img alt="" src="https://cdn.filestackcontent.com/rUj57NnGTi68BAh6PlEO"/>Anomali Match: Generating a Feed & Contribution Impact Report Step 3 4. Click Save and it is all set to run at the specified time.The report contains three sections of information that break down feeds and log sources in useful ways. For instance, one section displays the number of matches found and how many high confidence matches were found for each threat feed. It even highlights how many internal assets were affected by those matches. This information can help an intelligence program in a number of ways. Should a feed be tuned for better fidelity for your organization? Is there a configuration issue affecting feed performance?<img alt="" src="https://cdn.filestackcontent.com/1YheS9mMS6myPMqFDrO8"/>Anomali Match: High Confidence Feed MatchesLooking at the other data provided in the report from Anomali Match, we can see the logs that are doing the heavy lifting with regards to matches against threat feeds. We can also see which feed sources are being utilized the heaviest and which ones lack coverage and may need more visibility into.<img alt="" src="https://cdn.filestackcontent.com/lvfEgLT5RaOAO6DQMtcT"/>Anomali Match: Feed Matches by Log SourceAs seen in the first two screenshots, there is a feed called “Analyst.” By looking at this feed in the report, it will reflect how your organization is generating intelligence, what types of intelligence are created, and where that intelligence is used.And lastly, the report provides the threat categories that the various threat feeds show matches for. You can tell that some feeds may focus in on certain types of indicators whereas other feeds deliver several different types.<img alt="" src="https://cdn.filestackcontent.com/5SbyJDtWSa6CuQY8ZkO6"/>Anomali Match: Feed Matches by Indicator TypeIt's important to note that feeds are not a pure numbers game. Just because you brought in a million IOCs from a feed last month and saw 67 million matches doesn’t mean it’s doing an effective job for your organization. If most of those were false positives that simply filled your tools with noise then how useful is that feed as configured? Perhaps it’s just a case of tuning that feed to make more reasonable matches and suddenly the numbers are more realistic. More importantly, the feed is now more effective for the organization due to the tuning.Also consider that just because a feed only had five matches last month doesn’t necessarily make it useless. Maybe that particular feed doesn’t show matches very often - but when it does, it saves the organization from being front page news. That would definitely not be one you’d want to turn off.The other missing thing is institutional context. Are certain feeds constantly providing valuable insights when doing investigations? Maybe those matches weren’t high confidence, but the information they provide is valuable due to the additional context delivered.So keep all these things in mind when looking at the numbers in reports around threat feeds. This is not to diminish the value of reporting on feed matches and activity; it’s just to make sure we have the right perspective on what is being provided. The information from this type of report is there to provide insights into what is happening but it isn’t meant to tell the entire story. The remaining context will have to come from other sources: other tools that use the feeds, tickets or investigations that leverage details from the feeds, the relative value of the matches - which matches prevented attacks that would have otherwise been successful?What does all this information from the report tell us? The main point is that this is valuable information to help show how our threat feeds and log sources are being utilized in our organization. These insights highlight where remedial action may be necessary and give us measurable feedback on progress. The best thing about this is that the results displayed are particular to your environment, including impact on internal assets. The report also shows gaps that may exist in log collection and can measure progress in addressing those gaps. For example, logs that contain IP addresses but not domain names or file hashes will only be useful for feeds that at least have some IP addresses. So there may be few if any matches against that feed for that log source. Building out logs and feeds and ensuring proper alignment between the two while focusing on covering necessary areas of the infrastructure helps maximize the value from these resources. Lastly, this report also sheds light on the effects of tuning the feeds to limit false positives and deliver the best fidelity possible.