QOMPLX Knowledge: Responding to a Golden Ticket Attack

This is the latest in a series of posts we’re calling “QOMPLX Knowledge.” These posts are intended to provide basic information and insights about the attack activity and trends that are driving malicious campaigns and that QOMPLX researchers encounter in our forensic work with customers.

Here are the previous entries in our QOMPLX Knowledge series; look for more in our QOMPLX Knowledge series in the days and weeks ahead:

• QOMPLX Knowledge: Golden Ticket Attacks Explained
• QOMPLX Knowledge: Silver Ticket Attacks Explained

Recently we described Kerberos Golden Ticket attacks. These are a dangerous type of Kerberos ticket forgery in which an adversary gains control over an Active Directory Key Distribution Service Account (KRBTGT), and uses that account to forge valid Kerberos Ticket Granting Tickets (TGTs). This gives the attacker access to any resource on an Active Directory Domain (thus: a “Golden Ticket”).

Now that we have explained how Golden Tickets work, it’s time to talk about how to best investigate and recover from devastating attacks against Microsoft’s Active Directory platform like Golden Tickets. Here are the steps organizations should take to detect, analyze, and recover from Kerberos Golden Ticket attacks.

Key Points

• Basic security hygiene is critical in preventing attackers from gaining a network foothold and initiating a Golden Ticket Attack that facilitates privilege escalation and lateral movement. That means: keep Windows clients and servers up-to-patch, because most initial incursions exploit weaknesses for which there are already patches available.
• In the event of a Domain Control compromise, reset the KRBTGT service TWICE in order to generate a new signing key, and ensure the compromised key has been deleted.
• During recovery, parse and analyze Kerberos logs and Active Directory information to understand which network resources a threat actor accessed via the Golden Ticket, and what data may have been taken. Technology such as QOMPLX’s allows organizations to ingest, parse and analyze this data.  

As we discussed in our prior blog post, a Kerberos Golden Ticket gives adversaries the ability to forge new Kerberos Ticket Granting Tickets (TGTs) within a compromised Active Directory environment. Golden Tickets enable unfettered access to networked resources and allow an attacker to persist on a network indefinitely disguised as a credentialed administrator-level user.

Step 1: Prevention

Because Golden Ticket Attacks are post-exploitation attacks, enterprises' best and most effective defense against them is to block exploitation in the first place. The best way to do that is to promote and enforce basic security hygiene as a front-line defense. You should:

• Provide rigid security awareness training that teaches employees safe internet habits, including how to spot phishing attacks, where to report them, the business consequences of clicking on random links in email, social media, messaging applications, the risks of removable media, and more.
• Enforce the principle of least privilege for user accounts. Organizations should strictly limit the number of admin accounts; policy should dictate that user and local accounts have access only to those resources required to carry out their job functions. That same principle should also apply to applications, limiting their access to resources as well.
• Prioritize vulnerability and configuration management, and patch high-risk systems. Eliminating exploitable security holes is the best way to reduce the number of potential exposures that can be leveraged by an external threat actor or malicious insider.
• Reduce access to domain controllers. Read-only domain controllers can be used with clients in specific situations and will facilitate quicker recreation of the environment in the event of a Golden Ticket Attack.

Step 2: Detection

In the event of a compromise that leads to the creation of a Kerberos Golden Ticket, detection is difficult. Attackers who have forged Golden Tickets are indistinguishable from legitimate, credentialed users within a network. Detection requires organizations to weed out forged Golden Tickets from otherwise legitimate network traffic; a painstaking task, especially in large and heterogeneous networks. You must review all “suspicious” user account access to targeted systems, and then rule out the presence of malware, lax network or application configurations or exploits of vulnerable IT systems.

That’s why technology such as QOMPLX’s is critical to the detection of Golden Ticket. QOMPLX makes it possible to spot Golden Ticket attacks deterministically: ingesting and parsing critical logs and telemetry from Kerberos, servers, VPNs, and firewalls as well as intrusion prevention sensors (IPS) looking for indicators of these stealthy intrusions.

Golden Ticket 'Tells'

Sometimes Golden Ticket detection is straight forward. For example, attackers using tools like Mimikatz to forge Ticket Granting Tickets issued by the Kerberos key distribution center. Lazy attackers may opt to stick with Mimkatz default settings when generating their Golden Ticket or drop other “tells,” such as exceeding the default lifespan recommended for Active Directory (a maximum of 10 hours for a user ticket). That can be a useful heuristic in identifying a suspect TGT. However, sophisticated attackers will take steps to avoid giving themselves away, often opting for standard ticket “time-to-live” settings, the better to avoid detection.

Domain Field Irregularities a Clue

There are other clues that aid in the detection of Golden Ticket Attacks. For example, Sean Metcalf, founder and principal consultant at security consultancy Trimarc, points out that some third-party tools used to generate Kerberos TGTs may not format tickets exactly as Microsoft does in Windows. Metcalf says that a forged ticket may not may not properly populate the domain field in the Windows security event log. He said that the domain field could be left blank, or instead be populated with the fully qualified domain name (FQDN) or other values that would signal an anomaly.

What’s with the Weak Encryption?

Another signal would be the use of RC4 encryption of a Kerberos ticket. Modern networks, Metcalf suggests, would not use a broken cipher such as RC4, and today’s Windows would support reliable encryption ciphers such as current versions of AES. It should be noted that if an attacker encrypts a forged ticket with AES, this would be more challenging to detect.

Step 3: Response

Recent incidents have shown how compromises of domain controllers often precede devastating hacks, including deployment of ransomware and wholesale theft of data and intellectual property. That’s why, if detected or even suspected, a forged Kerberos Golden Ticket attack should trigger an immediate response from your security operations center (SOC), computer incident response team (CIRT), or third-party service provider. As part of that response, organizations will want to answer some form of the following questions during what are often catastrophic breaches enabled by Golden Ticket Attacks:

• How did the attackers initially access the network?
• What accounts were compromised by the attackers?
• Which IT assets have attackers accessed and/or compromised?
• What information have the attackers accessed and exfiltrated?

With a Kerberos Golden Ticket, attackers are able to access any data that trusts the domain that they have compromised. Investigators must assume that any data in a compromised domain could have been exposed, and then conduct a forensic investigation to determine which system(s) and data were exposed. That involves reviewing logs in order to identify compromised accounts and secure them. Investigators need to establish a clear access path that attackers followed through the organization.

Step 4: Re-image, or Watch and Learn

Once a Golden Ticket Attack has been detected and the basic dimensions of the compromise are understood, organizations face a choice: shut down affected accounts and take compromised assets offline to stop the attack, or hold back and observe the attackers at work.

Resources, visibility may limit choices

In QOMPLX’s experience, most compromised organizations choose the former; they shut down affected accounts, reset passwords, re-image systems, change remote access systems, and implement additional controls in the hopes of shutting down all avenues to future threat actors.  A rebuild may be an organization’s best option given available resources and visibility into such network activity. While understandable, this response eliminates the possibility of analyzing the attack and determining its full extent: what was accessed, and possibly who was behind the attack.

Observation carries risks

For organizations that choose to delay a response in order to learn about the compromise, technology such as QOMPLX’s can be used to observe the attackers as they move laterally on the network. The knowledge gained in observing attackers’ access to a compromised environment can be used to create events and rulesets as activity emerges. For example, victims can identify call-outs to certain command and control infrastructure, browsing of internal data stores, opening ports, accessing new accounts, and more. As this observation continues, behind the scenes, security organizations should also work closely with senior management, legal, and human resources teams during a compromise to prepare should the conditions change and a more active response becomes necessary.

No easy answers

By allowing an attack to continue, organizations accept short term pain (continued attacker access to their environment) for long term gain: better knowledge of how many accounts have been compromised; which systems those accounts have accessed; and whether the attack can be traced using logs and tools. Further observation might reveal how the attackers gained a foothold and what they were after; all of this better informs next steps for decision makers. However, there is no guarantee that victims will be successful in observing attackers: behavior might get missed, or attackers may become aware they are being watched and act accordingly: accelerating their attack and forcing the organization’s response. The choice to either respond or to observe isn’t binary and organizations should develop detailed contingency plans should their stealth observation fail and active response becomes necessary.

Step 5: Reset and Recover

Once the organization has a fuller understanding of the incident, it is well positioned to recover from it. This could include disabling all but a few of the lowest-privileged accounts, blocking external IP addresses associated with the attacks and, implementing redirects for suspicious traffic to honeypots.

By monitoring the few remaining active accounts, investigators could understand secondary attack paths used by the threat actors and whether they have dropped a backdoor into an organization that could be used indefinitely.

External validation of Kerberos

First, external validation of the Kerberos protocol is required to assure that every ticket presented by a Kerberos principal (i.e. service client) was in fact issued by a legitimate key distribution center. Microsoft has created a script to facilitate changing KRBTGT account passwords to minimize negative impacts.

Reset KRBTGT account

The next step in recovery is resetting the KRBTGT. For forged tickets, the KRBTGT service must be reset twice, once to generate a new key and a second time to delete the compromised key. It’s worth noting that if the attack paths used by a threat actor have not been addressed and shut down, resetting the KRBTGT account would be at best a temporary safety measure as an attacker could re-use their access to compromise other accounts and forge new tickets.

Rebuild and recover

Breached organizations may opt to immediately rebuild compromised systems, including resetting both end-user and service accounts and limiting access from outside in order to ensure continuity. Credentials and keys should likewise be reset for any “crown jewel” services or anywhere that evidence suggests a compromise has taken place. This is a disruptive, time consuming and costly process, but it can be a more favorable option than monitoring an attacker’s movements for a longer period of time, depending on data sensitivity, system visibility, and the ability to accept downtime.

The biggest challenge in this scenario is that it becomes nearly impossible to confirm how access was first achieved. As a result, recovery may not be complete at this point. Subsequent investigation should consider network assets such as VPN connections or local accounts that could be missed during cleanup. A thorough accounting of these and continued monitoring is needed to prevent an attacker from gaining a foothold and finding their way back into the network.

Additional Reading

QOMPLX Knowledge: Golden Ticket Attacks Explained

QOMPLX Knowledge: Silver Ticket Attacks Explained

How Active Directory Attacks went Mainstream

QOMPLX: The Importance of Lateral Movement Detection

Detecting Forged Kerberos Ticket (Golden and Silver Ticket) Use In Active Directory