QOMPLX Operations: Prioritizing Logs in Security Operations

For too many organizations, logs are a neglected source of valuable situational intelligence. Time and again, the presence (or not) of proper logs has been the difference between a successful defense and a significant breach or operational disruption within the targeted organizations.

That's why, for an incident response team, logs are the digital canaries on which the entire incident response plan relies. Too often during incident response or remediation, we find that the logs were overwritten; rolled over; not enabled or simply not stored and collected. Accurately piecing together and reporting the tools, techniques and procedures used by an adversary becomes exceedingly difficult, less accurate, and much more expensive.

If logs are not available, incidents response teams find themselves needing to focus on managing an ongoing incident at the same time as they attempt to engineer and implement a viable logging strategy to gain sufficient visibility. This wastes valuable time, reduces response efficacy, and ultimately increases costs during an incident.  In other words: the lack of logging is more costly in direct terms and in terms of indirect costs: prolonging the incident response and delaying recovery.

For an incident response team, logs are the digital canaries on which the entire incident response plan relies.

It is paramount for every organization to not only develop a prioritized logging strategy but also to consistently test and evaluate an adopted strategy against its goals. This should ultimately ensure they are able to perceive scenarios of concern and develop an associated threat model.

Visibility Matters

Log sources vary wildly and corralling them can be a challenge. Every organization must address operational challenges and limitations of logging if they are to come out ahead during an incident.  Visibility matters.  Cost control requires discipline. Utility requires more than just collecting and dumping log sources into storage or a SIEM.

A primary initial challenge is ensuring that logging is enabled and configured at the right level whenever possible. Do all your Windows systems have PowerShell logging enabled?  Does your Cisco ASA Firewall apply timestamps? Are DNS logs being collected?  Can I search across log sources?  Are data models unified?  Are log formats or settings being changed over time and - if so - by whom?

Check Logs Before You Need Them

Many people are surprised what systems do NOT log by default.   More and more systems only log metadata about actions taken. The majority of event data must be viewed in the console.  The take-away is to verify that what you think is in your logs is actually present, long before you need them.  

There is no substitute for inventory and preparation. In our experience, default settings usually require updates to ensure their utility to a centralized log management and search tool. Delaying or avoiding the hard work here could set you up for a disaster later on.

Essential Logs To Monitor

While one size doesn’t fit all when it comes to logs and log sources, there are some essential sources that should always be monitored regardless of your threat model and visibility needs.  Some of the logs that every IT organization should be monitoring are:

• Endpoint Logs: Endpoints, clients and servers, are often ground zero for intrusion events into an organization. It makes sense to mature your logging capabilities here. These logs are so valuable that attackers will spend a considerable amount of effort to tamper with these logs (disable or selectively delete). Today many mature organizations leverage the enhanced logging capabilities afforded by Sysmon from Microsoft’s Windows Sysinternals to extend these logs further. Recent versions of Sysmon include client DNS request logging capabilities which is strangely absent from Windows. A good starting point for a configuration is available at https://github.com/SwiftOnSecurity/sysmon-config
• Endpoint Detection and Response (EDR): While the EDR market has grown over the last few years to a multi-billion dollar offering, not all EDR solutions are built the same. An EDR solution should help your incident investigators quickly locate a threat and understand how that threat was delivered to your asset. Your EDR solution should support device isolation techniques that allow the EDR solution to remotely quarantine the asset so that the chain of attack can be broken, allow for indicators of compromise (IOC) hunting across the enterprise and offer an intuitive interface. A great EDR solution will achieve this while providing malware protection for devices regardless of the operating system.
• Cloud Service Provider: From IaaS, PaaS to SaaS, so many of our computing assets are being moved to cloud service providers. But did we remember to enable robust logging along the way? Can you tell if your HR manager is logging in from multiple locations, physically separated by thousands of miles, at the same time? When did they log on? Using what device? What files were accessed and by whom? Are they physically present in a facility during a VPN session? You will need to answer these questions and more if you are to successfully defend your cloud environment. A good cloud solution will enhance your logging capabilities and not sacrifice it along the way.
• Email Security Provider: While many organizations have great email transaction logging due to the Sarbanes-Oxley Act of 2002, many may lack forensic logging capabilities when it comes to malicious attachments and links. A great email security provider will provide a detailed log for each email and its associated threats. These logs would include sender, recipients, file hashes, IP addresses, disposition, malware analysis findings and threat campaign trending.
• Network Infrastructure: After so many years and dollars invested, it would be safe to assume that your network logging is world class. This may not be the case, however. Network infrastructure llke routers, firewalls and IPS/IDS solutions generate so many logs that tough choices have to be made as to which to keep. It is common, during an incident, to find that critical network logging was not accomplished simply because of storage concerns stemming from lack of planning.
• Multifactor Authentication: As multifactor authentication moves from "best practice" to "standard practice," it is becoming more essential for mature organizations to keep and store multifactor authentication logs as these multifactor systems as being more of a target for high stakes attackers including nation states. These systems should be logged and monitored as a high value asset.
• DNS Logs:DNS is a protocol that is at the heart of most networks. Despite being a foundational component of modern networking, DNS monitoring is a bit of a magic unicorn. Wouldn’t you want to know if your clients are calling out for a strange domain with 63 characters? With so many ways to abuse DNS, logging exclusively for DNS can be challenging. With that said, advancements in EDR & Firewall technology has made this a bit easier. Additionally, recent versions of Sysmon from Microsoft allow for DNS client requests to be logged at the client. By default, Sysmon will not log DNS client requests and it must be enabled.

Conclusion

Logs do not have to be a neglected or misunderstood source of operational insight or forensic intelligence. As our world becomes more complex, it is essential to protect and maintain your logs as you would any other high value asset.  

Do the basics well and then move on to higher-order rules, analytics, and ideas. Discipline here pays dividends for steady-state operations on an ongoing basis, but is especially critical when a true incident ultimately occurs.

This is the latest in a series of blog posts we call “QOMPLX Operations.” These posts are intended to provide security practitioners with best practices and insights needed to build effective, robust security operations center (SOC) teams. To learn more, download our free reports!