Red vs. Blue
Inside the world of the elite hacker and those trying to stop him
On any given day, in a city not far from Seattle, there’s a guy trying to gain access to Microsoft’s cloud. But he’s not just ‘a guy’ and he’s not just trying. He is a highly-skilled hacker leading an elite team of specialists who are persistent in their attempts to find a way in.
They do not give up.
Want to jump into this hacker’s thought process? Here it is: “We’re going to try to by-pass any protections that might be in place,” he says. “Anything to help us get at that goal.”
That goal is getting past Microsoft’s prevention systems. There are a lot of them. So our hacker is using every trick in the book as the cyber attack gains pace. Some of the tactics he mentions include credential theft, spear phishing against service operators, insider attacks, cross-site scripting, and other client-side attacks. You don’t need to be an engineer to know that this doesn’t sound promising for those of us with valuable private data stored in the cloud.
And of course part of any criminal activity is the challenge of not getting caught. “All the while we’re trying to stay forensically clean and remain undiscovered in the environment by covering our tracks,” he explains.
These are direct quotes. He is removing his fingerprints at the same time that he’s stealing.
How do you catch someone like that?
Here’s how: You have a team of people who can actually find a needle in a haystack. This particular towering haystack is the mass of data moving around every day. The needle is our hacker making one tiny slip-up and leaving evidence of his activity.
The lead security responder – the person chasing the hacker – describes his job like this: “What we do is intrusion detection so we are taking data from hundreds of thousands of machines in our datacentre, pushing that data onto the cloud, and looking at that data, at those log events, for anomalous behaviour.”
It’s about response: Speed. Agility. Accuracy. The security responder leads a team that may get a call in the middle of the night saying an alert has been triggered, saying an issue has been escalated to them. Saying it’s time to get to work. Doesn’t make any difference if it’s 3pm or 3am.
So here is where we stand: The hacker and his team are trying to breach Microsoft’s cloud infrastructure; the security responder and his team are trying to stop them.
Oh, and they’re all employed by Microsoft.
Red teaming: More than a war game
Hackers have names and faces. So let’s name ours. He is Travis Rhodes, Senior Security Lead for Office 365. He is the head of the Red Team. The in-house baddies. “You can think of us as an internal team of hackers focused on protecting Office 365,” he says. “We think and act like the adversaries that might attack our service or our customers. We analyse and probe the service for vulnerabilities, track the latest emerging threats and trends, to better simulate attack scenarios for Office 365.”
But here’s the twist. This is not a war game – this is not a situation where all players know ahead of time what is going to happen, where someone says ‘go’ at the start and ‘stop’ at the end. Because a war game is a fire drill. This is as close to a real fire without actually burning down the house (no customer data is touched – more on that later). And Travis’ Red Team has a large amount of creative freedom to attack Microsoft’s cloud and no one, that is no one in the security response team knows whether the attack is the work of Travis and his crew or if it is a real-world assault.
It’s the data protection equivalent of hiring a UFC fighter to attack you at random, unexpected times. Keeps you on your toes. And your heart in your mouth.
Because the Red Team doesn’t give up. Just like a real attacker.
So let’s say they get inside. Let’s say Travis and the Red Team manage to compromise the cloud. This is where all their research and preparation starts to pay off as they use the latest known attack techniques along with their own creative methods and custom-built tools. And mirroring a genuine attack, they change tactics as they go. It is a dynamic situation.
What next? Start small then go big. They acquire insider privileges and then use those to penetrate the infrastructure even deeper. Like someone stealing your library card but somehow managing to create a path all the way to your bank account. The whole time they do whatever they can to retain continuous access, to keep that foothold, while always trying to stay undetected.
And this is no leisurely stroll around hacker-land. The Red Team are racing against the clock. They are being measured first on how long it takes to compromise an asset, and then secondly on the time it takes to achieve full compromise. Those are two ominous-sounding words. Full compromise. Most organisations are not equipped to deal with a breach at this level. It can take several forms, and in our scenario could be the point at which the Red Team has acquired domain administrator privileges. It’s game over. Or game won. Depending whose side you’re on.
There are rules of engagement however. Let’s not forget that. The Red Team do not target customer data, nor do they ever interrupt the availability of the service or compromise in-place security. They are focused solely on attacking Microsoft infrastructure, platforms and applications – not end-customer’s applications or data.
Fighting back: Meet the Blue Team
“Everything that happens in our datacentre comes under my team’s microscope.”
This is Matt Swann, Senior Test Engineer at Office 365. We met him earlier, he was our ‘lead security responder.’ Matt is the head of the Blue Team, and they are in the business of classification. It’s through classification that it’s possible to categorise everything that looks like a piece of hay, and everything that might, possibly, potentially, be a needle. And then go find it. (Microsoft in fact operates dedicated full-time Red/Blue Teams for both Office 365 and Azure.)
Working with his Blue Team, Matt seeks to define what good activity looks like, and what bad activity looks like. Then they examine what falls in between. That’s where the job gets more complicated, and it’s where the Blue Team really earn their stripes.
“I use machine learning to build a statistical model of what my accounts, what my servers do,” says Matt. From there, he and his Blue Team start to delve more deeply: “I investigate the long tail, things which are anomalous, things that haven’t happened before and don’t seem to be happening anywhere else.”
It’s the sheer size of Microsoft’s operation that works to the Blue Team’s advantage here. Why? Because when you have a lot of data represented clearly, it’s actually easier to spot overall patterns. And in those patterns the Blue Team can look for any strange activity. “I have all of these servers globally in Office 365 and they’re all uploading security events like process starts [and] application events [as well as] engineer activity like logons and network activity all to a big data system in the cloud we call Cosmos,” says Matt.
So this is where, like detectives, the Blue Team can sift through the evidence. Checking those logons (ie, someone trying to access a system or application) and any other activity while they continually ask the question: Does this indicate that there has been a compromise and if so at what level? From there they can engage the most suitable engineers to work on the issue and assess the size of the breach. They then work on a plan to defend, evict and recover.
But we’re getting ahead of ourselves. It’s time to rewind, with Travis and his Red Team on the run in mid-attack. “When the Blue Team detects us that’s when the cat and mouse game really starts,” Travis says. “We try to stealthily access data while the Blue Team works to defend and kick us out of the service. The exercise is treated as a real incident until the Blue Team figures out it is us.”
This means it’s only during the post-mortem between the two groups that the Blue Team can accurately assess how successful they have been in challenging the Red Team’s attack. And with each go-round, the Blue Team further refines their approach and methods to protect Microsoft’s cloud infrastructure and keep customer data safe.
The best defence is assuming the worst
Red Teaming itself is based on a change in philosophy when it comes to protecting customer data – a change which surpasses current industry requirements and one which the company has been using for several years now. The strategy didn’t evolve from a breach of the Microsoft cloud but from this crucial observation: Many of the organisations being breached were unaware they had been compromised.
They didn’t know someone had got in. Often for several months.
Chang Kawaguchi, Group Engineering Manager for Office 365 Security, explains it this way. “We recognise that no computer system is perfectly secure,” he says. “So we invest heavily in an Assume Breach approach.”
Assume breach. Is this admitting defeat? It’s the opposite: It’s an acknowledgement that from time to time persistent adversaries are able to breach the cloud. This is a fact. We have all read stories about such incidents over the last few years, whether carried out by nation states or lone teenagers.
Assume Breach is a unique and proactive response to this reality.
It’s saying: Let’s assume the worst and act accordingly. Because even if you never get invaded, it’s nice to know the army is training for every possible scenario.
And when you place limited trust in everything – applications, services, identities, networks – you are never on the back foot. Think of it this way: When you view with suspicion any activity going on internally and externally, you are more likely to be able to respond faster and more effectively.
The fight continues
Cyber security is an ongoing battle. And the response must be relentless. Red teaming is only one part of Microsoft’s multi-faceted approach to cyber security, but in seeking to mirror real-world attacks it is at the frontline of the fight. It helps protect the Microsoft platform and keep your data and business safe, day and night. It’s not a replacement for prevention-based security, but simply represents an additional, complementary level of protection.
If you’re thinking of putting in an application to join the Red Team – well, good luck. Aside from the necessary expertise, each member goes through extra validation, background screening and of course training before they can get close to being involved in any attack scenarios.
So while you’re at work or at home and maybe wondering what’s going on up there in the cloud – spare a thought for the Blue Team, fighting invisible foes from around the world.
As well as other foes, just down the hallway.
Find out more, and download the Cyber Security Demystified eBook
SOURCES
