Clean-up On Aisle 2452: Weighing CISA’s SolarWinds Eviction Guidelines

The hack of SolarWinds and thousands of its customers may have faded from the headlines. But, off the front page, the work of assessing the damage from the now-infamous supply chain attack goes on. So does the work of evicting the hackers responsible for the attack, UNC2452, from compromised government and corporate networks.  

To that end, the U.S. government's Cybersecurity and Infrastructure Security Agency (CISA) published updated guidelines last week for companies that need to evict SolarWinds attackers from their networks. The guidelines are just the latest from CISA, which has issued a series of alerts, advisories and tools since soon after the SolarWinds hack came to light in early December, 2020. As more has become known about the incident, CISA’s advice has evolved from mere awareness and detection of the threat to increasingly detailed incident response and remediation steps.

The latest guidance, AR21-134A (PDF), provides the most detailed information yet on the steps needed to evict the SolarWinds actors from compromised environments and regain control of key IT assets. But the recommendations, which track closely to Microsoft’s own guidance, may conceal risks. Organizations should weigh their options carefully before plowing ahead with CISA’s recommendations, QOMPLX believes.

What You Need to Know

So what do you need to know about this latest CISA guidance? A few points stuck out to us here at QOMPLX.

Abandon All Hope All Ye With Admin Credential Theft

We wrote in March that the CISA guidance for SolarWinds victims wasn’t for the faint of heart. Basically, the agency tells the most affected organizations - Category 3 - to plan to sever their network from the Internet for the duration of the core eviction process, which could last days.

The updated guidance contains more sobering news for the most seriously affected SolarWinds victims. Specifically, for organizations whose investigation into the breach uncovers evidence that administrator level credentials have been compromised or for organizations that identify SAML abuse, CISA’s advice is to “consider the entire identity trust boundary compromised” and, given that “this threat actor is among the most capable,” should undertake a “full rebuild of the environment” from scratch. That’s strong medicine, but it's tucked into a side note in the guidance that might otherwise get overlooked. That is why we think it's worth noting up top.

Before Eviction: Scope, Harden, Cleanse

For organizations that aren’t in that dire situation (or that are, but would rather take their chances), CISA’s guidance on eviction outlines a multi-step process to both assess the impact of the breach and recover from it.

Victim organizations are instructed to determine the scope of the breach by identifying artifacts left behind by the attackers, telemetry from endpoints and networking equipment and other signs of unauthorized configuration changes.

Next, they are instructed to harden their network to outside access. That includes steps like removing Internet access from domain controllers, shutting off or monitoring external communications via SSH, SFTP, RDP, etc., auditing and culling unnecessary or ill-defined firewall exceptions and using host-based firewalls to thwart lateral movement.  

Finally, victim organizations need to clean up problematic cloud trusts between on-premises infrastructure and resources like Microsoft 365 and Azure. This involves looking for evidence of permission and credential changes to applications and service principles or suspicious or unauthorized modifications to federation trust settings. So too, cloud administrative accounts and applications. The focus of the assessment should be to determine whether the existing rights and credentials are “as intended” and necessary.

Eleven Step Program

The agency guidance lays out an 11 step eviction process.

Remove Threat Actors, Update Passwords

Victim organizations are directed to remove any known malware, backdoors or implants they may have detected during pre-eviction investigation. Passwords are to be rotated for any accounts used to manage network devices, including keys and strings used to secure network device functions like SNMP, IPsec/IKE pre-shared keys, routing secrets, TACACS/RADIUS secrets, RSA keys/certificates, etc.).

To limit the impact of account compromises, CISA’s guidance recommends organizations do a detailed audit of account privileges that were used on affected SolarWinds Orion servers, then create “unique and distinct administrative accounts” for each set of administrative tasks. Privileges and accounts that are not explicitly needed should be culled. Group policies that disable remote, interactive logins are encouraged as is the use of the Domain Protected Users Group.

Embrace User Least Privilege

Organizations are also encouraged to embrace best practices like user least-privilege and Just Enough Administration in rebuilding their environments. The use of multifactor authentication should be enforced for all administrative accounts and functions, as well.

With that accomplished, organizations can begin resetting passwords and secrets for their accounts. CISA’s guidance indicates that this should include password resets for all Active Directory accounts and service accounts, the Directory Services Restore Mode (DSRM) account on domain controllers, Windows local administrator accounts and privileged application accounts that are not part of Active Directory, such as privileged accounts on “high value assets.”

Cut Cloud Ties & Rebuild

Finally, the CISA guidelines provide detailed steps for restoring the integrity of Microsoft Azure and Microsoft 365 environments, starting with severing connections between those cloud-based assets and the on premises environment; severing federation trusts between on-premises networks and the cloud; and migrating to an external identity provider or to Azure AD to manage users.

For Microsoft 365, organizations are advised to isolate administrative accounts for M365 from any on-premises identity stores and create cloud-only administrative accounts secured with multi factor authentication and closely monitored. Tokens for any M365 accounts should be revoked and compromised mailboxes, tenants and Azure applications should be audited and cleaned.

Abandon Active Directory? Proceed with Caution

As we’ve discussed before: Active Directory is the biggest single cyber risk that organizations have and the highest value target of hackers. That’s why Active Directory compromises are a pillar of most sophisticated attacks. For example, we know that the attacks stemming from the compromise of SolarWinds Orion software played on many of the same weaknesses as earlier incidents, such as the attack on the Office of Personnel Management (OPM).

What Microsoft Said

In that light, it isn’t surprising that CISA’s guidance on evicting the SolarWinds hackers (UNC2452) is focused on reclaiming that asset from attackers, evan as it advises firms to move from on premises Active Directory to a more secure identity provider platform like Okta or Microsoft’s Azure AD. After all, Microsoft has been walking away from on-premises Active Directory quickly, beginning just around the time the SolarWinds breach came to light. (Coincidence? We think not.) In December and January, for example, the company announced it was retiring the Enhanced Security Admin Environment (ESAE) architecture (often referred to as “red forest,” “admin forest,” or “hardened forest”) which was the recommended approach to providing securing on-premises Active Directory. (See “Microsoft to CIOs: Drop Dead.”)

That approach is echoed in CISA’s guidance. The agency recommends “avoiding federated enterprise environments whenever possible” and refers readers to Microsoft documentation on protecting Microsoft 365 from on-premises attacks. That document advises Microsoft customers with federated environments that use both on premises Active Directory and Azure AD and Microsoft 365 to decamp for the cloud. Specifically, Microsoft advises organizations to “use Azure AD cloud authentication to eliminate dependencies on your on-premises credentials.” Organizations should provision users from “cloud HR apps to Azure AD,” thus ensuring that “an on-premises compromise (will) be isolated, without disrupting your joiner-mover-leaver cycle from your cloud HR apps to Azure AD.”

While the Microsoft guidance doesn’t explicitly instruct customers to abandon on premises Active Directory, it certainly subordinates it to Azure AD. This includes having users “mastered” from Azure AD, creating cloud-only administrator accounts, and using Azure AD rather than on-premises Active Directory to provision access to cloud applications, external (third party or contractor) identities and so on.

Proceed with Caution

But is that the right move for organizations at this time? Here at QOMPLX we worry that by adopting Microsoft’s plans for addressing the issues raised by SolarWinds whole-cloth, CISA has missed an opportunity to do some independent and differentiated research and thinking about how organizations can best secure critical identity stores going forward.  

For one thing, by centralizing an entire identity management function on cloud-based identity platforms, whether Microsoft’s or another company’s, organizations are concentrating risk, not attenuating it. Microsoft’s Azure AD environment benefits from the full capabilities of Microsoft’s security team. But, at the end of the day, it is exploitable as well. And, unlike on-premises Active Directory environments, there will be no "RESET KRBTGT" available if- or when- Microsoft’s Azure AD environment is compromised.

In fact, Microsoft itself was a victim of the SolarWinds attack - and was slow to disclose its own exposure to it. Both those facts should give pause to organizations that want to assume that the Redmond company’s cloud-based infrastructure is impenetrable and that Microsoft has its thumb firmly on top of everything that transpires within its own corporate environment, let alone the massive Azure cloud environment.

The Takeaway

CISA’s guidelines for evicting the SolarWinds hackers (UNC 2452) are required reading for anyone with direct exposure to the threat - whether Category 1 or Category 3. Broadly speaking: they sketch an outline of security in the “post SolarWinds'' world. That sketch is of a world in which organizations implement strict control over user accounts, and also move rapidly away from on premises Active Directory and toward independent cloud-based identity providers like Okta, Azure AD and others.

While cloud-based IDPs offer many administrative advantages over locally managed identity infrastructure, we think that an abrupt change from one to the other introduces risks that aren’t well covered by CISA’s guidance. Not surprisingly, there is no “silver bullet” to stop attackers as sophisticated as those who compromised SolarWinds and its customers. Before making any major change to its identity infrastructure, organizations should consider all the options available to them. These should include ways to mitigate the risk to existing, on-premises IT assets, options for incident response and recovery following an attack and contingency plans for if (when?) a cloud based identity provider might fall to a sophisticated threat actor. Thus far, CISA’s guidance has not weighed in on these questions. CISA also doesn’t touch on non-trivial security concerns introduced by wholesale outsourcing of IDP responsibilities.

CISA’s guidance also makes clear how organizations - whether wrapped up in the SolarWinds incident or not - need to take a hard look at their existing and planned cloud investments and take seriously the risk posed by cloud based applications and services, as well as federated connections to existing, legacy infrastructure.

The days of magical thinking about the security of cloud environments are drawing to the close. So too the days of quick and dirty integrations from on premises to cloud environments. Going forward, harder lines are going to need to be drawn around cloud access and provisioning. Taller walls will need to be built to make sure that insecurities and breaches of legacy infrastructure don’t “federate” unauthorized access and cyber risk, as well.

That aside, the guidance for evicting the SolarWinds threats underscores what QOMPLX considers core capabilities and best practices that organizations should invest in and prioritize. These include: patch and configuration management; the use of multi-factor authentication to prevent easy account takeovers;  the application of threat intelligence and threat monitoring to focus detection and prevention efforts; improved monitoring of both endpoint and network activity to spot suspicious activities; and the use of strategies like least privilege, network segmentation and endpoint detection and response (EDR) to  identify compromises and stop lateral movement.

While not perfect, in other words, CISA’s latest guidance is required reading for anyone with even tangential exposure to the SolarWinds incident, and recommended reading for everyone else.