Doing Threat Intel the Hard Way - Part 2: Capturing Threat Intelligence

Part #2: Capturing Threat Intelligence

This is the second post of a series on manual management of IOCs for threat intelligence.

Part 1: Manual IOC Management

Once you have settled on the sources you wish to collect, a method, or more frequently methods, of collection must be established. If you have lots of sources identified, you are likely to be forced to support several different methods of collection. In some cases, delivery will be automated, such as TAXII over email, or received by email, but in a format that must be converted such as a csv, pdf, xml, or even free text. Some web sites will publish threat intelligence in HTML or XML formats, from which users may either capture it manually or script an automated method to scrape the site at a predetermined interval. STIX and TAXII are widely supported standards for formatting and delivery, but support is by no means universal.

An API (Application Program Interface) may be available for some feeds. This is particularly the case for most commercial feeds but may or may not be the case with open source or free intelligence feeds. The API’s themselves will generally require reviewing reference documentation to understand how to access them, how to request and/or retrieve data, as well as limitations on use such as rate limits. Leveraging API’s to ingest feeds can be fairly straightforward but does require scripting or some other mechanism to actually pull the data and do something with it. Additional care and feeding may be required over time as API’s do change as features are deprecated, new features are added, and tweaks are made for improved efficiency. Major overhauls of APIs are not unheard of and may break a lot of automation if previous APIs are deprecated. Monitoring API sources for updates is an important part of keeping feed collection running smoothly.

The main consideration for capturing threat intelligence is automation. You should automate as much of the collection process as possible. This can be done mostly via scripting but may require some additional efforts around collecting via email or web scraping. Putting in this effort pays off over time as manual collection consumes time few teams have to spare. It also frequently takes analysts away from their primary duties while they focus on the mechanics of manual collection. Source selection itself may end up being limited due to the inability to regularly capture the available data without it being manually collected.

Up next in the series: Processing Threat Intelligence