Tuesday, March 14, 2017

The Origin of Threat Hunting

2011 Article "Become a Hunter"
The term "threat hunting" has been popular with marketers from security companies for about five years. Yesterday Anton Chuvakin asked about the origin of the term.

I appear to have written the first article describing threat hunting in any meaningful way. It was published in the July-August 2011 issue of Information Security Magazine and was called "Become a Hunter." I wrote it in the spring of 2011, when I was director of incident response for GE-CIRT. Relevant excerpts include:

"To best counter targeted attacks, one must conduct counter-threat operations (CTOps). In other words, defenders must actively hunt intruders in their enterprise. These intruders can take the form of external threats who maintain persistence or internal threats who abuse their privileges. Rather than hoping defenses will repel invaders, or that breaches will be caught by passive alerting mechanisms, CTOps practitioners recognize that defeating intruders requires actively detecting and responding to them. CTOps experts then feed the lessons learned from finding and removing attackers into the software development lifecycle (SDL) and configuration and IT management processes to reduce the likelihood of future incidents...

In addition to performing SOC work, CTOps requires more active, unstructured, and creative thoughts and approaches. One way to characterize this more vigorous approach to detecting and responding to threats is the term “hunting.” In the mid-2000s, the Air Force popularized the term “hunter-killer” for a missions whereby teams of security experts performed “friendly force projection” on their networks. They combed through data from systems and in some cases occupied the systems themselves in order to find advanced threats. The concept of “hunting” (without the slightly more aggressive term “killing”) is now gaining ground in the civilian world.

2013 Book "The Practice of NSM"
If the SOC is characterized by a group that reviews alerts for signs of intruder action, the CIRT is recognized by the likelihood that senior analysts are taking junior analysts on “hunting trips.” A senior investigator who has discovered a novel or clever way to possibly detect intruders guides one or more junior analysts through data and systems looking for signs of the enemy. Upon validating the technique (and responding to any enemy actions), the hunting team should work to incorporate the new detection method into the repeatable processes used by SOC-type analysts. This idea of developing novel methods, testing them into the wild, and operationalizing them is the key to fighting modern adversaries."

The "hunting trips" I mentioned were activities that our GE-CIRT incident handlers -- David Bianco,  Ken Bradley, Tim Crothers, Tyler Hudak, Bamm Visscher, and Aaron Wade -- were conducting. Aaron in particular was a driving force for hunting methodology.

I also discussed hunting in chapter 9 of my 2013 book The Practice of Network Security Monitoring, contrasting it with "matching" as seen in figure 9-2. (If you want to save 30% off the book at No Starch, use discount code "NSM101.")

The question remains: from where did I get the term "hunt"? My 2011 article stated "In the mid-2000s, the Air Force popularized the term “hunter-killer." My friend Doug Steelman, a veteran of the Air Force, NSA, and Cyber Command, provided a piece of the puzzle on Twitter. He posted a link to a 2009 presentation by former NSA Vulnerability and Analysis Operations (VAO) chief Tony Sager, a friend of this blog.

July 2009 Presentation by Tony Sager
In the mid-2000s I was attending an annual conference held by NSA called the Red Team/Blue Team Symposium, or ReBl for short. ReBl took place over a week's time at the Johns Hopkins University Applied Physics Lab in Laurel, MD. If you Google for the conference you will likely find WikiLeaks emails from the HBGary breach.

It was a mix of classified and unclassified presentations on network defense. During these presentations I heard the term "APT" for the first time. I also likely heard about the "hunt" missions the Air Force was conducting, in addition to probably hearing Tony Sager's presentation mentioning a "hunt" focus.

That is as far back as I can go, but at least we have a decent understanding where I most likely first heard the term "threat hunting" in use by practitioners. Happy hunting!


Unknown said...

I have been publicly talking about 'hunting' since 2006. For example in this presentation: http://www.slideshare.net/zrlram/rsa-2006-visual-security-event-analysis

I didn't call it 'hunting' but 'exploration', but one of the first references to the process of hunting?

Anonymous said...

Do you consider that honeypots or new techniques using Deception approach fit into Hunting techniques or NSM tools? Because the idea is to catch to the attackers into the traps or lures to determine their actions/TTPs.

Richard Bejtlich said...


Hunting is a process. The data source can be varied. If you are hunting using data from deception technologies, then I think the process applies.

Tony Sager said...

Richard: Concerning the origins of cyber "Hunting". First, I am *honored" to be named as "a friend of this blog". Your writing has always been a must-read for me. Second, I am impressed that our mutual friend Doug remembered my presentation. Here's some more context for that diagram. It looks a bit thin because the real in-house version was "For Official Use Only", mostly because specific program/project names were included. And that acronym in the upper right is Advanced Network Operations, the organizational name of the Hunt mission that Doug heard about.

My intent was to show a unified model of all of the jobs done in my Group at the time - VAO (Vulnerability Analysis and Operations) - and how they were all evolving in 2 dimensions: Horizontally, from discrete events (e.g., A Red Team exercise), to connecting discrete events to draw conclusions about the state of security across the DoD ("sampling"), to an ongoing ("persistent") search for vulnerabilities and attackers. And Vertically, from the identification of vulnerabilities in lab/products to the operational world, to the observation of them in real-life, to true adversary emulation (context: most penetration test, Red Teams are/were a poor emulation of real Adversaries), to an active and sustained search for Attackers ("Hunting").

For us, "Hunt" meant a very planned and sustained search, taking advantage of the existing infrastructure of Red/Blue Teams and COMSEC Monitoring, as well as intelligence information to guide the search. That mystery phrase on the lower right ("Persistent but not Pervasive") was meant to say a lot - we were aiming for very large reach, but you cannot look everywhere all the time, so you need to guide the hunt. I honestly don't recall the first use of "Hunt" in this context, but I am sure it did not start with me in 2009.

So bottom line: "Hunt" emerged as part of a unifying mission model for my Group in the Information Assurance Directorate at NSA (the defensive mission) in the mid-late 2000's. But it was also a way to unify the relationship between IA and the SIGINT mission - intelligence as the driver for Hunting. The marketplace, of course, has now brought its own meaning to the term, but I just wanted to share some history.

--tony sager
CIS - the Center for Internet Security
(retired from NSA in 2012)