How to Spot and Stop Attacks Like SolarWinds [Video Interview]

Don't Replace it, Repair it.

Watch our latest video interview with Baird to learn how to protect and preserve your Active Directory today.

The QOMPLX CYBER software suite detects and prevents Kerberoasting and Golden Ticket attacks, both of which appear to have been used in the hack of U.S. federal agencies that was announced last week.

In our latest interview with Baird, the global financial services firm, we discuss why these types of breaches are so bad for companies and how firms can recover quickly from such a hack.

Video transcript

[COLIN]

One. Alright, thanks everyone for joining us today. I'm calling Sebastian, Internet Analyst here at Baird, and joining me as host on the call are my colleagues Rob Oliver and Jonathan Rickeyver, who cover software and security for us. And my distinct pleasure to introduce Jason Crabtree, who's the co-founder and CEO of QOMPLX, one of the leading private companies in the space of AI, data Intelligence and cybersecurity. Jason has a very long and distinguished resume, which will be hard for me to do justice to you today, but he's a West Point grad, a Rhodes Scholar with an engineering degree from Oxford. He led infantry in Afghanistan. He graduated from Ranger School. And I got to say, it makes me feel very good knowing that there are people like Jason running companies and hopefully helping agencies and other businesses deal with these external threats that are hard for those of us on the outside really to comprehend. So Jason, thanks so much for joining us.

[JASON]

Really nice to be here.

And for those dialing, just as a reminder, feel free to submit questions to the webcast chat feature. You could also email me at csabasian@rwbaird.com, we'll try to get to a few of those. And for disclosure, just please refer to the event confirmation email, published research, or Baird’s website for important disclosures regarding any companies that get discussed during the event. So Jason, tell us a little bit about QOMPLX so we can put a little more context around your perspectives here today.

[JASON]

Sure. QOMPLX is about a six year old cyber security and technology company that focuses on really using data to get to ground truth for risk. And we're particularly advanced in large scale active directory and authentication related attacks. And we do work in that for some of the world's largest corporations. Some of the large companies that you would recognize are some of the largest technology firms in the world, some large law firms in the world, the largest title insurance companies, mortgage underwriting processors work in cyber insurance. So some of our executive team hails from the former Head of Cyber Risk at JPMorgan Chase, the former CEO Validus Reinsurance, the former CEO of JLT Towers on the reinsurance side. And we focus in particular at how risk and money come together with, again, a unique focus on what we would call anthropogenic perils. So cyber security, terrorism, those types of issues where you've got a living, breathing, thinking adversary coming for you and it's a little bit different than thinking about hurricanes or natural calamities.

Thank you, that's very helpful. So maybe just to start off at a higher level, as kind of a level set, can you put in the general terms what happened, who is responsible, and how bad really is the situation? I hear things like golden tickets.

[JASON]

Yeah, sure. I think it's important to break these attacks down into a couple of distinct phases. The reality is that you certainly have what we'll call the initial entry point phase. Right. In this particular case, it was the compromise of basically software that was provided by SolarWinds, which runs on top of networks. In this particular product, the Orion product was installed by about 18,000 customers that actually installed that infected software out about 30,000 customers that use that product. And that's clearly a problem, right? It's an initial entry point. It puts something that's malicious on a trusted device inside your corporate network. The next phase of the attack, though, is pretty different. The next phase of the attack is effectively not unique to anything about Solar Winds. And I think this is something that people really need to remember when they're thinking about it. Right? The supply chain entry point was just that, it was an entry point. It could have been achieved by a public facing Internet vulnerability. It could have been achieved by someone in an HR department clicking and downloading the same kind of malicious software, and the next phase of the attack would have been almost identical. Regardless, in this phase of the attack is what we call lateral movement. And lateral movement is effectively where you try and escalate privileges and move from wherever you got a foothold to where you wanted to go as an adversary in that particular stage of this attack. And if you can go through and see some of the commentary on this from NSA, from CISA, really involved exploiting protocols and tools like Active Directory to do things like Kerberosting or Golden Ticket attacks to actually gain these administrative rights. And once you have the administrative rights, you have the keys to the castle out on the network. If you're a domain administrator, you can add and remove users, you can issue yourself credentials, you can create new users. Now you're in the third stage of the attack. Now, if you go back, the same kinds of attack techniques were actually used by OPM from the Chinese in 2014. So that middle stage of the attack, actually we didn't learn much because it's been years and years and years and years, and the same kinds of techniques were actually used at OPM. The third stage of the attack is new, and it's something though, that CyberArk and QOMPLX started writing about in 2017 and 2018. So again, this is not a new idea, and it involves basically forging SAML tokens. SAML is effectively an authentication protocol like Kerberos, but it's very common in cloud types of environments, right? So this is how anyone who's ever said, let Microsoft or Google or Facebook serve as your identity provider for the most part on the Internet, that's SAML on premise in a corporate network. That's typically done with kerberos. Well, those SAML forgeries actually are really dangerous because what happens is, once they have those domain administrator rights in the core of your network, they actually stole the signing certificate and they could just issue themselves cloud login tokens, these SAML tokens, so that they could actually impersonate a user, not just in the corporate network, but in Federated cloud services. And so CyberArk actually released some tools to do that in 2017. It was called Golden SAML. QOMPLX also showed how you can compromise these things. And again, you can compromise Okta, you can compromise Auth Zero, you can compromise Microsoft, you can compromise anyone who uses the SAML protocol. It's not specific to any of the vendors that are there. And once you have that key material, or for most corporations that actually connect their SAML servers to their Active Directory environment, if you compromise Active Directory, you can actually use that Federated link to take over basically the next part of the chain. And that's why you're seeing NSA and others saying, hey, you run SAML services, which most large corporates do. You basically can't trust that anyone who used the SAML token to say this is me, I'm logging in, is real. It could have been the adversary, it could have been a real user. And the reality is, once you forge those tokens, you don't know them.

[COLIN]

So it's fair to say that the adversary that they're still present in these networks, they still have access, and if so, what does it take to get them out?

[JASON]

Yeah, so one of the reasons why these types of techniques are so bad is and frankly, it's why my co-founder Andrew Sellers and I left the government service to actually go build a company to work on detecting them. You actually can't even validate the logs you use to authenticate traffic and actually say who did what to whom on a network are real. When you spoof authentication, every single tool, security tool and network management tool says whoever used that authentication token is the user. Well that's a really big problem because it means your logs can lie to you, right? So you can't do behavioral analysis on this. You can't do any of these things that you typically would want to see because some of that forensic evidence doesn't exist and just collecting the log data actually isn't enough. Because SAML and Kerberos are actually protocols. And what that means is in the post forensic sense, there's actually no custody or ledger that actually said, here's the list of all the transactions that Colin or Jonathan or whoever's ticket were used from. Because on big networks there are multiple domain controllers which control Kerberos tickets and there are multiple SAML servers. And any one of those SAML servers or Kerberos domain controllers can issue you a ticket. So it's basically like going up to a hotel desk or something else, right? Anyone at the concierge can give you a key that will work for your room and you don't have to talk to the one person who signed you in. Anyone who has the approval authority in your hotel can issue you a key. Same exact thing on the enterprise network. So when you talk about expelling an adversary from one of these networks. The real issue is you actually have to stop them from forging these new credentials, otherwise they get to persist as an administrator. So until you regain control over SAML or you regain control over Active Directory, you basically can't trust any of the login events and that's why you're seeing FireEye and other experts coming out and saying hey, these networks are going to be burned to the ground. What that commentary means is if you own authentication, you own everything. So you have to basically reestablish trust that every event or log on event is real before you can begin to evict them or it's just whack-a-mole, they get to move around the network and continue to create or impersonate users.

[COLIN]

Before I pass it over to Jonathan, I do want to ask you quickly because I follow the big tech companies, amazon, Google, and given the Microsoft situation here, office 365 and Azure potential exposure, what do you think was the role of Azure in this infiltration? And secondarily, do you have any thoughts personally on if this affects the pace or the extent of migration to cloud platforms?

[JASON]

Yeah, that's a great question. I think two comments I'll make about it. The first is that Solar Winds and other vendors that sell software for critical infrastructure and companies clearly need to make sure that they have software development, lifecycle practices and others that are best of breed. And I do think that there is a challenge with supply chain security. Certainly there's lots of work to do, there. More forms and compliance artifacts that already exist that ask you to check some equivalent of have you been compromised by nation state today? Are not particularly helpful in doing that. So I think it's really important to talk about the fact that a Solar Winds-like breach could have happened to hundreds of other companies. And Solar Winds was targeted for some reasons about what their products do, but many companies would not have detected the kinds of things that impacted FireEye or Solar Winds in this particular case. The second piece is that the reality is the actual target organizations didn't have technology that allowed them to actually detect these kinds of authentication forgeries. So if you look at what Microsoft's response to this has been, they've said a couple of things. They've said first of all, everybody should buy Microsoft's Defender Identity Project, right? And that's a product that frankly QOMPLX competes against. And we've replaced in some of the world's largest companies because we have a superior offering. So Microsoft looks at behavior around ticket utilization for attacks like Kerberos, it doesn't actually keep a list of all of the Kerberos transactions that occur. Why? Because it's really hard and it took tens of millions of dollars to do and it took us a lot of time, but we were able to replace migrants on these large corporate networks where they said hey, we don't want to just look at behavioral artifacts of suspicious tickets. We want you to keep track of every ticket or every key basically that's ever been issued. And we want you to check every time someone shows a key that it was duly issued and authorized. So my firm is the only firm in the world that does that. And that's something that was, I'll call it hard one. Microsoft is actually trying to and it used to have something called Red Forest. So Microsoft had its Enterprise Security Guidance which was to go basically create these very secure on place for managing Active Directory on premise. And this was going to get peered or federated to Azure Ad. And again, you could just peer or Federate to JumpCloud or Google's Active Directory offering or Amazon's Active Directory offering or a variety of others that were doing that. Specifically how people even connect new companies like Op Zero and they use ADFS. And some of these techniques Microsoft has now come out this last week, right as this breach broke and they did something to their customers, what they came out and said was actually, we're no longer going to recommend that you run on premise Identity infrastructure at all. We think you should just all move to Azure Active Directory. Let's talk about what that means. When you actually have your own Active Directory administration for your company, not when you use one of these pure cloud-hosted environments. You own the root of trust. You own the master key, right? And that's the Kerberos ticket granting ticket. You control it. When you bind a computer to your Active Directory instance and you are a big business or a government agency and you control that, you can reset it, you can manage it. You are a distinct thing. It's what Microsoft calls a true security boundary. When you are in Azure Active Directory, the binding of a computer to Microsoft is not binding it to your unique trust. You're bound to Microsoft's root of trust. If you want to talk about a calamity waiting to happen. It's all these corporations moving to trust Microsoft as the root of trust, where if you compromise Microsoft, you compromise every single user of Active Directory globally. By the way, Microsoft acknowledges in its new announcement that it does not even follow its new guidance. It continues because of its actual focus on providing hosted services to implement its prior guidance, which was to implement Secure Enclaves for Active Directory. It just says you shouldn't do it. You should just use our hosted services because it's easier for you, even though that's not secure enough for us. And I think that that's the kind of thing that corporations should be watching out for. And it's literally a huge supply chain issue that's much more significant, frankly, than there. And Microsoft quote in its announcement from this says “while Microsoft no longer recommends an isolated administrative forest model for most scenarios at most organizations, microsoft still operates a similar architecture internally and associated support processes and personnel because of the extreme security requirements for providing trusted cloud services to organizations around the globe.” So it's what they do, but they are now telling customers, just buy more of our stuff and move to the cloud, even though we don't do that ourselves and wouldn't actually be able to operate to our own security standards for hosted operations if that was the case. So honestly, cloud services are a great thing. We wholeheartedly recommend them to a lot of folks. But making the cloud your identity provider, you better understand exactly what you're getting and you better be able to understand that you're not trusting their trust. And trusting their trust is different than trusting something that you actually hold the master key to.

[COLIN]

Okay. With that, let me pass it to Jonathan who has a couple of questions and he'll pass it along to Rob after that.

[JOHN]

Thanks Colin. Jason, thank you very much for doing this. A lot of questions in terms of how this event pushes priority within security spend near term and obviously FARA is on the front line in terms of incident response and forensic work. But a lot of speculation out there that EDR, XDR technologies like SIMS can help identify or detect this type of activity and prevent it. But it's clear listening to you, that you don't think that's the case and you do mention privilege to comp security and Cyber Arc is potentially doing this, but help us understand: rank the top three priorities of your sizeo that you would be looking at to help better protect against these types of issues.

[JASON]

Yeah, I think there's a couple of things that you have to focus on. So first of all, let's disabuse folks have a couple of common myths here. Behavioral analysis relies on the faulty authentication data to attribute a given action to a given machine or behavior. So if you can't trust authentication, then I can't say it was you Jonathan, versus it was Colin. So the moment someone starts to tell you they're going to use a statistical analysis about what your normal behavior is when someone can attribute traffic to your account or to a different account. Let's go back to first principles. The second piece is when you start looking at the data types for this and this is that post forensic comment I made earlier. The only way to administer and actually track Active Directory transactions is your soft defender for what it does or what QOMPLX. Lots of companies have very basic, and this is included SIM providers have very basic abilities to do very novice things around Windows event logs, which is a great form of telemetry, but it's easily bypassed. Windows event logs should be high on lots of corporations lists and they can do real time analysis to look at brute force password detections. They're a part of Kerberosting or Ntvsd extraction or a bunch of these common lateral movement techniques, but they don't actually stop the golden ticket attacks, which allow you to become a domain administrator. They don't allow you to stop silver ticket attacks to do that, either defender for Identity or QOMPLX Identity insurance solution actually take every single Kerberos transaction off the wire at the domain controller or at the services. All those other tools that you talked about, your XDR, your EDR, our providers, they literally don't have the data feed. So if somebody is telling you they can do that, there's a reason why Microsoft or why we talk about these very specialized instrumentation techniques to do that. So that's something that you've got to do. The second piece, when you look at it, is zero trust architectures, right? So zero trust architectures have kind of been ruined by marketing departments everywhere. But the reality is, zero trust architectures are really supposed to mean like an egg with a heart, right? The moment you get in, it's all yoke. So when you think about zero trust architectures, what you really want to be doing is you want to say that authentication is actually done at the service that you want to talk to, right? So don't just VPN into your corporate network, and then you can access everything. You want to have very narrow access and ideally terminate basically encryption directly at a given service. So you're doing a point to point connection between any of your users or any of your services. So that's a great thing to work on for organizations, and it's something that needs to be at the top of their list. The third one is more automation for analysis of, like, Credential usage. You really want to be looking proactively at privilege management. Privilege management is really, really hard. The press has talked for a long time about vulnerability management, but folks, vulnerabilities have never been the goal. Privileges are the control. You don't want to hack in if you can log in. And so the reality of this is the whole goal of being an attacker on your network is to become appropriately authenticated traffic. As soon as I'm authenticated, I'm real. And so when you think about what you want to do, is you want to make it really clear to understand what we call the blast radius of an account. If you have a highly privileged user that has administrative rights to everything, that's probably a bad idea. If you have administrators that you don't understand or maybe browsing the internet as an administrator, that's an awful idea. If every one of your employees has local rights, can install things on their computer again, that's an awful idea. And so some companies, ours included, build tools that help connect to your sale point, privilege management system or other kinds of things, and they allow you to look at that data in a way that shows you how you would move laterally and use those privileges to attack the organization. So when you go back to Sale Point, you go back to one of these Pam tools, you're able to then update how privileges are allocated to users. And you're going to reduce the impact of a Phishing attack or of a service getting compromised like Solar Winds on a specific box. But that's the top of the list. If you've got folks saying that they're going to do this stuff with Sore tools or with SIM tools, they literally don't connect the data to the actual issue here. So that's folks getting outside of the sensor data that's available to them to establish ground truth.

[JOHN]

Right, that's helpful. So if you are in the Chief Information Security Officer that had an issue with the Orion download to contain that malware, what would be the three initial steps you'd be taking to identify and then hopefully contain and then prevent on an ongoing basis what measures, what technologies?

[JASON]

Sure, yes. The first thing that you're going to want to do is to actually take a look at things like DNS and Exfiltration logs. Right. So DNS is really helpful. You can look at your perimeter logs and you can see whether or not any of the beacons that were expected were effectively there. And FireEye, you have to give a tremendous amount of credit to Kim Mandya because FireEye did a fantastic job of actually publishing some of the known C nodes. So got your computers talking to some of the bad C Two. That's the first thing on saying, hey, did I see this back to Dlly? The second piece you want to look at is you need to go to Active Directory and you need to reset that root of trust directory. That's called resetting. KRBTGT. It's the Kerberos ticket granting ticket. When Dr. Seuss weighing the core from your network, that is the core Credential that is actually used underneath everything. So that's the certificate. And so what you're going to do is after you check C Two to see if anything beacon out, you need to go straight to Active Directory and you're going to roll that Credential twice. And once you roll that Credential twice, you're going to start to look at basically going to any of the services that any of the machines, basically that ever had that malware installed on it that might have been malicious or that was malicious. And you're going to start by looking at service accounts. And service accounts are typically where adversaries will go for Active Directory movement because they're stealthier than user accounts and people don't look at them as often. One of the common attacks that was used in this case was actually Kerberosting. So that's basically a way of escalating up to gain administrative privileges from using that service account to get additional material off the DC. And they're going to use a Kerberosting attack to try and get credential information. And if they're able to get Credential information, then they'll do what are called ticket forgeries to become a domain administrator for Golden Ticket Attack Kerberosting, you can partially detect with Windows event logs. So that's something you're going to want to get administered if you can. And you're going to want to look at suspicious administrative uses or privilege elevations. But if you actually see anything there you really need to do is you've got to start instrumenting your domain controllers, and you need to be looking at where those service accounts and what else was running on those machines. So that's the thing. Start for C2 reset the core of trust in your network by rolling care BTGC. Start with service accounts and administrators, and then you need to go out and look at whether or not Active Directory has any suspicious administrative behavior. And then you need to check whether or not Active Directory was configured and peered to your SAML environment. And if it was, you need to be looking in particular at your Azure Tenant or any of your cloud tenants where that SAML federation occurred to see if the credentials from Active Directory could have actually in fact, been used to manipulate or modify your cloud tenant environments. If you do that, you're going to be on your way. The last thing I'll say, Jonathan, I think this is really important. When you have these kinds of breaches where effectively your main administrative accounts were compromised, cleanup does not look for the binary, and hey, it wasn't there. Oh, I saw my DNS logs. By the way, most companies don't store twelve months of DNS logs. So when you have people coming out saying there's no evidence of compromise, a lot of them don't necessarily even have the data to make that statement. So you've got to go back at least September of last year on DNS logs now, and you need to be looking at any of the other edge logs, and you need to be looking at your administrative rights and Credential history. And if you can clear all of that, you still need to be doing what we would call a Hunt operation to basically look at suspicious service accounts and kind of proactively identify things. The last thing you can do, if you're doing all of that, then you can say, hey, let's go ahead and create some really juicy fake accounts. That might be the kinds of things, when these guys went to ground and they want to come back, they're going to reset their access and probably do the same attack again. So you're going to want to create some older, what we would call Honey accounts or Honey tokens. So like some fake service accounts that look really old and look like they've never been actually maintained, because that's the first place they're going to go to launch another Kerberosting attack in the future to come back and regain their administrative rights as they try and reestablish themselves in your network.

[JOHN]

Yeah, no, that sounds like a very painstaking process. My last question is just how long does it take a large company to be able to adequately go through that assessment? Does it potentially put a pause on other IT projects within a large enterprise?

[JASON]

Well, I think the reality is when you have a compromise of your Active Directory environment and when you have evidence, like in the SIS advisory of things like Golden Ticket Attacks or the SAML signing certificate or if someone has KRBTGT, it literally means that the attacker can do anything they want on your network. So you don't have a more important priority than stopping this. And the challenge for that, for a lot of these organizations is for organizations that don't have DNS logs, for organizations that haven't already either implemented QOMPLX Identity Assurance Piece or Microsoft's Defender for Identity, they actually didn't even collect some of the Kerberos data or authentication or log on data that they need to expeditiously do the breach detection. You won't get it from Crowd Strike or from Silance or a number of these EDR solutions and vendors because they actually don't pull that information. And so one of the challenges that organizations are going to face here is it's a BB King quote, right? The most important note is the one that isn't played. The most important node is the data that you didn't save. And so for organizations that could do this, they actually have to go run these prolonged Hunt operations. And I would tell you, for most organizations that didn't have any significant evidence, they didn't see a bunch of those known C Two addresses from Beacons inside their network calling out to C Two. They didn't have any of those things. They're probably looking at a Hunt operation where a couple of people are looking for bad guys in their network by pivoting through some of their common areas for three to four months. If they did have that stuff, and there is evidence that they had administered a Credential compromise that's there, it's going to be much, much longer than that and it's going to be a pretty expensive process.

[JOHN]

Okay, that's helpful. Let me pass it to Rob for any questions he has. Great.

[ROB]

Thanks, Jonathan. Jason, really appreciate it. I know we're bumping up against time deadline. A lot of questions have come in from Bear's clients as well. So I'm going to try to squeeze some of those in and combine them with questions that we had as a team. But you mentioned supply chain entry point, and I think that struck me because I know a lot of our clients are invested in those supply chain companies, which in many cases are either infrastructure companies or application software vendor companies that sell to your clients. So I guess the first question that I would have in the Aggregating, the questions that have come in here is like, what does that mean for those merchant vendors of software and on the application side, what is going to be required of them, if anything, to change in order to be compliant in a new environment. And then obviously, everybody's worried about Solar Winds. I know you made it clear that this could have happened with any vendor, but just the extent to which the SolarWinds Orion platform will be able to recover from this. And it was a lot in there. Thanks.

[JASON]

Yeah, sure. So maybe let's go with that last one first, which is to say to SolarWinds, right? Because the SolarWinds product and platforms do a lot of them in the network and there are a limited number of commercial alternatives for large enterprises. And frankly, SolarWinds has provided a really critical service that is economics extraordinarily attractive. I think Solar Winds is likely if they do the right thing and given their responsible ownership, I'm sure they will over time. They have every ability to invest appropriately in SDLC and other controls to make sure that they're able to field that product in a way that's there if you replace SolarWinds with someone else, you're trusting that they put in all these software development lifecycle controls that SolarWinds may or may not have had or may or may not have been effective. But you're making the same bet. People should not be thinking that SolarWinds is unique in gaps around this. I've worked with SolarWinds on all kinds of federal and commercial networks, and the reality is, I think most customers are probably going to be best served by sticking with it and working through them to actually solve the problem. The folks that are out there saying that my company would never have been impacted by this kind of supply chain attack had a nation state targeted them are frankly pretty arrogant, in my view, around thinking about how complete and comprehensive their own controls are. By the way, that's true for the big cloud providers as well. And this is part of my comment about saying you're better off occasionally having a problem like a SolarWinds on your network and having a single entity like Microsoft or Google pull all the master keys for hundreds of companies. That's a much scarier problem than it is for thinking about having an administratively credentialed device somewhere on your network is literally not owning your network. So when you move a little bit deeper into that question set and you say, well, what happens for some of the other security tools and priorities? Right. One of the things that I hope folks are asking these firms that make mission critical software is, hey, do you actually do anything to secure authentication and do you actually do anything to secure privilege management? Privilege management programs are extremely difficult to run well, and that's often why they haven't been done. But everybody's made noise for a very long time about vulnerability management, even though vulnerabilities were never the goal. Vulnerabilities were just the entry point. Every single attack on a corporate or government network ever. Its goal was to get administrative credentials so that they could become a legitimate user and take actions as an authenticated member of the network. So privilege management and identity assurance is much, much more important than volunerability management, always has been. But that was something that has never, I'll call it, gotten the attention that it deserved, even though it is the root of trust. And then what was the third one, Rob? You have to remind me.

[ROB]

No, I think you answered they were both in there. It was about just did the application vendors and so I think you got it. A couple other questions, Jason I'll try to get them in here. The federal government, this is always interesting for me to say because we don't think of the federal government as leading, but federal government going back to the 21st Century Idea Act and the Federal Data Strategy, and even a few weeks ago, President Trump signing the Centers of Excellence law. Federal government has really led on cloud. And we have a FedRAMP program which is supposed to certify software companies, software companies tell investors how proud they are to have gotten FedRAMP certified and that gives them license to come in and sell. It's a good housekeeping seal of approval. So I guess, broadly speaking, what, if anything, and I heard you say that this is your top priority right now, but what, if anything, does this do to that process? Do we see projects slow as we handle this particular problem? And how might that Fed authentication process change, if at all?

[JASON]

Yeah, I mean, I think there's a couple pieces here. The first of it, which is neither the federal government nor the majority of corporations have had effective privilege or authentication management or controls really ever. And the reality was that that wasn't what was. People were getting burned in part because these things are more complex than that, is really kind of a rebranded diet cap, which used to be an older standard around that. And the federal RMF process is actually pretty lagging behind. And this is one of the problems with what we'll call prescriptive solutions here, right? Prescriptive solutions from clients perspective is we're going to score you about how well you implement controls against the attacks that used to happen. So over the past five or six years, even though these authentication attacks have been widespread since 2014 and by the way, are a part of Merck, Merisk, FedEx, North Kydro, all those mass ransomware attacks you've been reading about, those are also facilitated attacks. It was difficult and it was a lot easier to say, hey, my software will never so maybe they'll never get to that stage. My firewall provider says they'll never get in. My scanning provider says we'll always catch the vulnerabilities first. But the reality is that when you hear people use words like “assume breach” what assume breach really is supposed to mean is that it doesn't matter if it's SolarWinds, it doesn't matter if it's the dancing hamster or lol cat that you clicked on. It doesn't matter if it was a targeted phishing attack. If you want to phish any corporation in the world, send out an Excel document with a compromised macro that says year end bonus plan, I guarantee people will click that right and they install themselves. It didn't matter there was SolarWinds anymore because you just invited the same people to put the same kind of binary on your document. And the reality is there's enough ways to get behind your basically endpoint protection device that you can reliably plan on defeating your EDR provider. And that means you go back to attacking Active Directory and you're back to attacking authentication. So if you do nothing else and we already saw the world's best companies, I think, starting to do this. We work for one of the world's largest technology investors and firms, and they disqualified Microsoft solution in favor for us two years ago. One of the largest corporate networks in the world, we do a billion transactions a day, validating Kerberos traffic on their network to try and help them prevent these types of attacks from occurring. And they knew that this stuff was coming and they said, we have to do golden ticket attack protection, we have to do silver ticket attack protection. A lot of companies looked at that and said it's just too expensive, it's too hard. We'll just hope and it's been in the hope bucket for far too long.

[ROB]

Got it. One last question I want to squeeze in because it's coming from a bunch of clients and just specifically relative to the fixes that SolarWinds has put out there. I've spent some time on the chat boards over the past week getting a sense from, you know, what people think about those. And talking to clients is your sense that those hot fixes, hot patches that they put out, have they at least fixed the issue for now, for those who have SolarWinds?

[JASON]

I am not the best person to talk to about the completeness of SolarWinds response. I have not personally evaluated the binaries they've provided. I will say for some of the corporations we work with that are Solar Winds customers, by and large, they have been happy that the SolarWinds appears to be fixed and we do see them actually coming on some of those recent hot fixes out there. The reality is SolarWinds was compromised and we still know the details there. We're going to want to know more before declaring victory. They're built servers were compromised, we know that. But I think we've just got to trust the SolarWinds team to work with their customers on actually fixing the original entry point again. I think for enterprise security folks and for investors though, the takeaway is SolarWinds isn't the story. The supply chain part of it is sexy and it makes people think of James Bond movies and whatever it's going to be. But that's actually not the issue because a phishing attack could have functionally done the same thing. When you actually think about getting malicious software onto the network, that's not the part that really got us here, and that's not the part that made it stealthy. The part that made it stealthy was that we didn't detect anyone doing lateral movement after the initial entry vector.

[ROB]

Got it. Well, we have a bunch more questions. Colin, maybe I'll turn it back to you for some final words.

[COLIN]

Thanks, Rob. Well, Jason will be respectful of your time, but since I'm the high level question guy and Rob and Jonathan have all the intelligent, specific questions, maybe just for me, eight months it took to figure this out, or at least discover the infiltration, why did it take so long? I mean, is there any way to calculate what the potential damage could be? Could adversaries have access to or sensitive national security data? How should we think about that?

[JASON]

Yeah, the unfortunate truth of this is there's a reason why you don't want to give everyone in your organization administrative rights. And that same reason applies to why you don't have someone who's not a member of your organization with administrative rights. Right. Organizations realistically, especially given the timelines that are here and the fact that a lot of these companies won't have even external perimeter logs that would have shown exfiltration of data over the entire period over which they were likely compromised, it's going to be very difficult in the reality is we may never know exactly what was taken. There is no post-hoc analysis here that is going to have all the data that's there. And so what I would say is that companies and investors would be well advised to avoid these declarative statements that say there's no evidence of a breach. Well, congratulations. We didn't put a camera on that part of the till and we didn't see anyone taking money, and we don't know how much money they put in the till, so there's no evidence of a breach. Right. I think that's the problem with some of the analysis and some of the things you're already seeing. And this is the point about identity infrastructure. Identity infrastructure is the keys to the kingdom. You have to defend it at all costs because we call it the Apex control. Every single IT network management and security control in the entire organization relies on it. If it is compromised, none of them function as intended. And so the frustrating part I'll share. Andrew Sellers, our CTO was at the Air Force and I was on the army side. But we personally briefed the Defense Department department about these exact kinds of attacks over six years ago in 2013 and 2014. And the government has resisted going ahead and actually investing in these capabilities because it was hard and because they frankly listened to some of the vendors that were out there that were saying, hey, we'll just catch on the end point, we'll just catch in these other places. And that was always not a good idea. So I do think that there's a lot of, I'll call it false indignation about how unforeseen this was, where even the SAML Forgeries and ADFS attacks Cyber Arch and QOMPLX have been publishing about that publicly since 2017 and 2018, showing active directory takeoffs to take over ADFS Federated services. And then Cyber Arch did some awesome research showing SAML Forgeries if you compromise such certificates. So this is not a new thing and we should stop pretending it is.

[COLIN]

All right, well, Jason, we'll wrap it up with that. Thank you so much for your time. This has been incredibly helpful and we'd love to get back with you. Maybe in a couple of months when we think settled down a little bit, we have a little bit better view into what the next steps are.

[JASON]

Hey, thanks, Colin. Rob and Jonathan. Really nice to be here and appreciate you involving me. Cheers.