Three months after the SolarWinds compromise first came to light, the U.S. Government’s lead cyber security agency in the last week has released a new tool for scanning compromised environments, and a detailed remediation plan for federal agencies caught up in the sophisticated, Russian-orchestrated hack. It’s not pretty.

The Cybersecurity and Infrastructure Security Administration (CISA) published the CISA Hunt and Incident Response Program (CHIRP) on Thursday. CHIRP is described as a forensics collection tool that can “help network defenders find indicators of compromise (IOCs) associated with the SolarWinds and Active Directory/M365 Compromise.” CISA published the free tool to the Agency’s GitHub repository.

CISA’s remediation guidance, released on March 9, calls for the agencies most affected by the SolarWinds hack to disconnect their enterprise networks from the Internet for “several days” in order to properly evict the SolarWinds actor, regain control over their Active Directory environment and rebuild affected systems. It is intended for federal agencies, but is also applicable to private sector firms caught up in the operation.

Categories 1 and 2: The SolarWinds Bystanders

The Agency’s guidance is geared to organizations with different levels of exposure. For Category 1 and 2 organizations, which either did not use Orion, did not detect any evidence of compromise or that detected an initial compromise, but didn’t see evidence of lateral movement, CISA offers a standard menu of security best practices: up to date antivirus signatures and engines, patching of vulnerable systems, enforcing strong password policies and user education to spot phishing emails.

Organizations in Category 2 are urged to do all those things, but also continue monitoring for possible “follow on adversary activity” and signs of compromise within their environment. Any government organization that used the Orion product is expected to conduct analysis of system memory, host storage and network and cloud forensics looking for indicators of compromise (IOCs) and secondary “actions on objectives” (AOO). Any threat actor controlled accounts or persistence tools should be removed.

Category 3. See also: Category 5

The story is different for so-called “Category 3” organizations, which were the most severely affected by SolarWinds. For those organizations, CISA’s guidance suggests that nothing short of a complete rebuild of their enterprise network environment is likely to evict the attackers.

Unplug the Network

Among other things, CISA instructs Category 3 organizations to “disconnect from the Internet for 3-5 days” while they engage in “resource intensive and highly complex” remediation activities. “Failure to perform a comprehensive and thorough remediation will expose enterprise networks and cloud environments to substantial risk of long-term undetected APT activity, including email monitoring and data collection and exfiltration,” the agency warns.

Remediation for Category 3 organizations comes in three phases: Pre-Eviction, Eviction and Post-Eviction.

Pre-Eviction: All Eyes on Active Directory

In the Pre-Eviction stage, organizations need to identify a “trust boundary” - essentially, the enterprise assets that need to be scrutinized and investigated. Category 3 organizations then need to conduct a thorough investigation of suspicious account activity associated with SolarWinds Orion servers and service accounts used by Orion. Any credentials stored on the SolarWinds server are especially suspect and Category 3 organizations should assume they are compromised - possibly for months.

Organizations are advised to audit all network device configurations stored or managed on the SolarWinds monitoring server or other network devices for signs of unauthorized or malicious configuration changes. That includes “any local configurations that could be loaded at boot time.” CISA advises Category 3 organizations to turn their endpoint detection and response (EDR) products ‘up to 11’ and do “aggressive collection,” especially on sensitive assets and endpoints.

Also as part of the Pre-Eviction investigation, organizations need to look into abuse of federated services using SAML forgeries. (See our QOMPLX Knowledge piece on Golden SAML attacks.) In general, CISA warns that finding evidence of SAML forgeries is a worrying sign. “If the adversary has compromised administrator credentials in an environment—or if organizations identify SAML abuse in the environment—simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA notes in its remediation guidelines. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate.”

Finally, organizations are expected to migrate on-premises resources to an Azure AD, cloud based identity infrastructure—consistent with guidance Microsoft released at about the time the SolarWinds compromise was revealed. Beyond that, Category 3 organizations are instructed to closely audit permissions and credential changes for applications and service accounts, with special attention to overly permissive applications, unusual credentials or modifications to federation trust settings. The same is true of Microsoft’s M365 tenant configuration.

As QOMPLX CISO Andy Jaquith noted in his post “Microsoft to CIOs: Drop Dead,” the company’s guidance on using Active Directory increasingly looks like an ‘orderly retreat’ rather than a plan for victory. This has been echoed by other security experts, who note a building crisis of confidence in Microsoft as an identity provider - which is concerning given that Microsoft owns the lion’s share of the enterprise identity market. QOMPLX believes that new tools are needed to monitor Active Directory for sophisticated attacks. We also recommend a number of steps organizations can take to shore up the security of legacy Active Directory environments, including domain consolidation.

Going forward, and considering the pain and resources required to conduct the kind of ground-up rebuild that CISA outlines (and that Microsoft has hinted at) QOMPLX advises its customers to seek an alternative means of securing their identity infrastructure.

Eviction: Mop and Monitor

Phases 2 of the remediation process is focused on “evicting” the threat actor and asks Category 3 organizations to “regain sole control over their AD, remove malware implants from network and cloud systems and rebuild or e-image network and cloud systems.” Easier said than done; CISA promises specific guidance to agencies on how to accomplish these tasks.

Post-Eviction: Maintain Vigilance

In Phase 3, the “post eviction” phase, Category 3 organizations are asked to make a comprehensive report to CISA and to “maintain vigilance”: staying on the lookout for “known TTPs” associated with the SolarWinds actors and “signs of persistence,” including command and control (C2) connections to new domains and attempts to run “unusual code.”

Conclusion:

Attacks and threats affecting Microsoft’s Active Directory and Kerberos identity infrastructure are reaching crisis proportions. In the midst of this crisis, however, organizations can’t afford to panic, nor can they stick their heads in the sand and ignore SolarWinds or other incidents.

QOMPLX has long advised its customers to be attentive to the risks posed by stolen credentials, privilege escalation and surreptitious lateral movement leveraging Active Directory. While Microsoft may be urging a flight to the cloud (specifically: their Azure AD identity cloud) the reality is that most firms will need to continue to support business-critical workloads that rely on on-premises Active Directory, even while they lay the groundwork for a post-AD future using Azure AD or some other cloud-based identity platform.

Continuing to support on premises Active Directory doesn’t mean business as usual, however. Your organization needs better tools and methods to detect lateral movement of attackers within your environment. That means more detailed logging of network, server and host activity and frequent auditing of your Active Directory environment. QOMPLX helps its customers with just these problems, using a range of tools and detection methods, including the application of streaming analytics to provide near real-time detection of lateral movement.

If you want to learn more about how QOMPLX can help your company spot signs that may signal an Active Directory or authentication compromise, contact our team now.