Lessons learned from the Microsoft SOC—Part 3d: Zen and the art of threat hunting

Threat hunting is a powerful way for the SOC to reduce organizational risk, but it’s commonly portrayed and seen as a complex and mysterious art form for deep experts only, which can be counterproductive. In this and the next blog we will shed light on this important function and recommend simple ways to get immediate and meaningful value out of threat hunting.

This is the seventh blog in the Lessons learned from the Microsoft SOC series designed to share our approach and experience from the front lines of our security operations center (SOC) protecting Microsoft, and our Detection and Response Team (DART) helping our customers with their incidents. For a visual depiction of our SOC philosophy, download our Minutes Matter poster.

Before we dive in, let’s clarify the definition of “threat hunting.”  There are various disciplines and processes that contribute to the successful proactive discovery of threat actor operations. For example, our Hunting Team works with threat intelligence to help shape and guide their efforts, but our threat intelligence teams are not “threat hunters.”  When we use the term “threat hunting,” we are talking about the process of experienced analysts proactively and iteratively searching through the environment to find attacker operations that have evaded other detections.

Hunting is a complement to reactive processes, alerts, and detections, and enables you to proactively get ahead of attackers. What sets hunting apart from reactive activities is the proactive nature of it, where hunters spend extended focus time thinking through issues, identifying trends and patterns, and getting a bigger picture perspective.

A successful hunting program is not purely proactive however as it requires continuously balancing attention between reactive efforts and proactive efforts. Threat hunters will still need to maintain a connection to the reactive side to keep their skills sharp and fresh and keep attuned to trends in the alert queue. They will also need to jump in to help with major incidents at a moment’s notice to help put out the fire. The amount of time available for proactive activities will depend heavily on whether or not you have a full-time or part-time hunting mission.

Our SOC approaches threat hunting by applying our analysts to different types of threat hunting tasks:

1. Proactive adversary research and threat hunting

This is what most of our threat hunters spend the majority of their time doing. The team searches through a variety of sources including alerts, external indicators of compromise and other sources. The team primarily works to build and refine structured hypotheses of what the attackers may do based on threat intelligence (TI), unusual observations in the environment, and their own experience. In practice, this type of threat hunting includes:

• Proactive search through the data (queries or manual review).
• Proactive development of hypotheses based on TI and other sources.

2. Red and purple teaming

Some of our threat hunters work with red teams who simulate attacks and others who conduct authorized penetration testing against our environment. This is a rotating duty for our threat hunters and typically involves purple teaming, where both red and blue teams work to do their jobs and learn from each other. Each activity is followed up by fully transparent reviews that capture lessons learned which are shared throughout the SOC, with product engineering teams, and with other security teams in the company.

3. Incidents and escalations

Proactive hunters aren’t sequestered somewhere away from the watch floor. They are co-located with reactive analysts; they frequently check in with each other, share what they are working on, share interesting findings/observations, and generally maintain situational awareness of current operations. Threat hunters aren’t necessarily assigned to this task full time; they may simply remain flexible and jump in to help when needed.

These are not isolated functions— the members of these teams work in the same facility and frequently check in with each other, share what they are working on, and share interesting findings/observations.

What makes a good threat hunter?

While any high performing analyst has good technical skills, a threat hunter must be able to see past technical data and tools to attackers’ actions, motivations, and ideas. They need to have a “fingertip feel” (sometimes referred to as Fingerspitzengefühl), which is a natural sense of what is normal and abnormal in security data and the environment. Threat hunters can recognize when an alert (or cluster of alerts/logs) seem different or out of place.

One way to think about the qualities that make up a good threat hunter is to look at the Three F’s.

Functionality

This is technical knowledge and competency of investigating and remediating incidents. Security analysts (including threat hunters) should be proficient with the security tools, general flow of investigation and remediation, and the types technologies commonly deployed in enterprise environments.

Familiarity

This is “know thyself” and “know thy enemy” and includes familiarity with your organization’s specific environment and familiarity with attacker tactics, techniques, and procedures (TTPs). Attacker familiarity starts with understanding common adversary behaviors and then grows into a deeper sense of specific adversaries (including technologies, processes, playbooks, business priorities and mission, industry, and typical threat patterns). Familiarity also includes the relationships threat hunters develop with the people in your organization, and their roles/responsibilities. Familiarity with your organization is highly valued for analysts on investigation teams, and critical for effective threat hunting.

Flexibility

Flexibility is a highly valued attribute of any analyst role, but it is absolutely required for a threat hunter. Flexibility is a mindset of being adaptable in what you may do every day and how you do it. This manifests in how you understand problems, process information, and pursue solutions. This mindset comes from within each person and is reflected in almost everything they do.

Where any threat analyst (or threat hunter) can take a particular alert or event and run it into the ground, a good threat hunter will take a step back and look at a collection of data, alerts or events. Threat hunters must be inquisitive and unrelentingly curious about things—to the point that it bugs them if they don’t have a clear understanding of something. Instead of just answering a question, threat hunters are constantly trying to ask better questions of the data, coming up with creative new angles to answer them, and seeing what new questions they raise. Threat hunting also requires humility, to be able to quickly admit your mistakes so you can rapidly re-enter learning mode.

Threat hunting tooling

Threat hunting naturally pulls in a wide variety of tools, but our team has grown to prefer a few of the Microsoft tools whose design they have influenced.

• Advanced hunting in Microsoft Threat Protection (MTP) tends to be the go-to tool for anything related to endpoints, identities, email, Azure resources, and SaaS applications.
• Our teams also use Azure Sentinel, Jupyter notebooks, and custom analytics to hunt across broad datasets like application and network data, as well as diving deeper into identity, endpoint, Office 365, and other log data.

Our threat hunting teams across Microsoft contribute queries, playbooks, workbooks, and notebooks to the Azure Sentinel Community, including specific hunting queries that your teams can adapt and use.

Conclusion

We have discussed the art of threat hunting, different approaches to it, and what makes a good threat hunter. In the next entry, we dive deeper into how to build and refine a threat hunting program. If you are looking for more information on the SOC and other cybersecurity topics, check out previous entries in the series (Part 1 | Part 2a | Part 2b | Part 3a | Part 3b| Part 3c), Mark’s List, and our new security documentation site. Be sure to bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us at @MSFTSecurity for the latest news and updates on cybersecurity.