Chapter 5
Containment
Doing the best at this moment puts you in the best place for the next moment.
—Oprah Winfrey
Learning Objectives
Understand the goals of containment, and recognize why containment is important
Know strategies for making effective decisions in the containment phase
Learn tactics for reducing the risk of data exfiltration, halting file encryption/deletion, and resolving denial-of-service attacks
Gain familiarity with techniques for rapidly locking the adversary out of the victim’s environment
When a cyber extortion attack hits, you need to act quickly and contain the damage by halting any malicious activities such as encryption, unauthorized access, lateral movement, data exfiltration, denial-of-service attacks, command-and-control communications, and more. At the same time, responders must ensure that the adversary is locked out of the environment as quickly as possible.
All of this must occur as soon as possible—often before responders clearly understand the scope of the compromise or have actionable intelligence about the adversary. Actions taken (or not taken) during the containment process can have a major impact on the speed at which operations can be resumed, as well as the risk of a data breach or other damaging consequences.
In this chapter, we review common containment strategies, including methods for halting ransomware encryption/file destruction, stopping data exfiltration, resolving denial-of-service attacks, and locking out the adversary. Along the way, we discuss mistakes to avoid and provide tips for successfully containing damage.
5.1 The Need for Speed
The goals of containment depend on the specifics of each cyber extortion situation. Common containment activities include the following:
Halt ransomware encryption/data destruction if it is still ongoing.
Disable persistence mechanisms that can automatically relaunch ransomware or other malicious processes.
Halt data exfiltration if it is in progress.
Resolve denial-of-service attacks, restoring normal access to data and resources.
Lock out the adversary: Ensure that all remote access mechanisms employed by the adversary have been blocked, so that they can no longer gain unauthorized access to network resources.
Hunt for threats: Identify and eradicate lingering signs of malicious activity within the network, such as persistent backdoors or signs of malware.
During containment, a seemingly small mistake can have dire consequences. If one of the steps is performed incorrectly, all the work performed during the other steps can quickly become irrelevant. Often, an error at this point will result in the following potential outcomes:
The adversary reenters the network.
More data is exfiltrated.
Encryption software re-detonates.
Data is lost or destroyed.
Critical evidence is overwritten.
In contrast, effective containment can save the organization from disaster, and even make the entire event seem relatively inconsequential. Quick and wise action can result in the following outcomes:
Expedient resumption of normal operations
Reduced risk of future damage or reinfection
Minimal need for legal or regulatory response
Limited media attention, which is aligned with the organization’s chosen message
Tip: Don’t Stomp on the Crime Scene
When a cyber extortion event is discovered, responders usually want to jump into action to contain the incident and get the organization back to business as usual as quickly as possible. However, to protect the organization, responders need to take care not to unwittingly destroy valuable evidence. See Chapter 6 for more details on evidence preservation.
5.2 Gain Access to the Environment
Before responders can secure the victim’s environment, they first need to gain access—not always an easy task when an adversary has taken over! During the lead-up to a cyber extortion event, an adversary will often take advantage of established remote access methods to expand their own foothold into the environment.
Responders need to ensure that authorized parties have the access they need to remediate the incident, while simultaneously removing the adversary’s ability to derail recovery efforts. Typically, responders use a combination of physical and remote access to accomplish these goals.
When possible, it is simplest for responders to begin with physical access only. With this approach, all remote administrative access can be blocked until responders are finished locking out the adversary (see Section 5.7).
If immediate remote access is necessary for responders, be sure to prioritize the steps in Section 5.7 right away, and restrict remote access as much as possible while still enabling responders to work. This can include allowlisting only known “good” IP address ranges for remote responders, increasing password strength requirements, setting a strong account lockout policy, enforcing multifactor authentication on all accounts, and taking similar actions to lock down remote access as much as possible in the early phases of the response.
Heads Up! Endpoint Detection and Response Software
During the initial containment of a cyber extortion attack, endpoint detection and response (EDR) software can be extremely useful for accessing the victim’s environment and containing the compromise. EDR functions by combining features from multiple types of security software, including anti-virus, intrusion detection/prevention systems, vulnerability scanners, and more. It also provides responders with a centralized system that can monitor and respond to potential threats to the network quickly and efficiently.
In general, EDR tools can be used to accomplish multiple tasks, such as the following:
Root out malicious behavior
Identify and block malicious software
Quickly isolate and quarantine critical hosts
Trace and eliminate suspicious system processes
If EDR tools are not already available in the victim’s environment, responders may want to deploy them? (i.e., tools) as a first step, which then facilitates the remainder of the response. See Chapter 10 for more details about EDR software deployment.
5.3 Halting Encryption/Deletion
In “denial” cyber extortion attacks, the adversary typically encrypts or deletes data to limit the availability of the information. Possession of a decryption key or a copy of the deleted files gives the adversary leverage over the victim.
In attacks of this type, the most time-sensitive priority is to stop any active file encryption or deletion processes. Quick action in either case can enable defenders to save some of the victim’s files or, at the very least, reduce the risk of repeat encryption during the recovery process.
Even if the malicious activity appears to be complete, it is best to treat any affected system as if the ransomware was still alive, well, and running—because it just might be.
Here are four strategies for halting malicious activity and preventing it from reoccurring:
Change permissions
Remove power
Kill the program that’s encrypting the files
Disable persistence mechanisms
In this section, we discuss the tactics, benefits, and potential pitfalls of each of these approaches in turn. Any method may result in some data loss, so be prepared for that risk.
Tip: Free Decryption Utilities
At the onset of an incident, first responders such as IT staff may panic and attempt to decrypt files using any public decryption utility they can find. This practice can have serious consequences!
Unless the decryptor is specifically meant for the exact variant and version of ransomware on a system, it will fail to decrypt anything, and it may corrupt the data that has already been encrypted. Moreover, many public “decryptors” have been discovered to actually be ransomware, leading to a double-encryption scenario.
In 2020, the criminal organization behind the Zorab ransomware strain began launching and distributing fake decryption software that was advertised as being able to decrypt STOP ransomware, a popular variant at the time. However, instead of decrypting files, the fake decryptor would add a second layer of encryption to files on already encrypted systems, making victims purchase two individual decryptors instead of one.1
1. Lawrence Abrams, “Fake Ransomware Decryptor Double-Encrypts Desperate Victims’ Files,” Bleeping Computer, June 6, 2020, www.bleepingcomputer.com/news/security/fake-ransomware-decryptor-double-encrypts-desperate-victims-files/.
5.3.1 Change File Access Permissions
A quick approach to stop encryption can be to change file permissions. It may not be realistic to recursively change permissions on every file in a large filesystem, but large assets such as database files can be quickly modified to prevent encryption. This is also very effective on shared drives when it may take some time to track down the source of encryption. As a first step, simply restrict share access to only a small set of trusted users and change files to read-only (and also to immutable on Linux/UNIX). If you determine that an admin-level account is being used, adjust the account so that it cannot be used to reset or override the filesystem permissions.
Tip: Plan for Cleanup
When attempting to halt file encryption, thoroughly document any file permission changes. These changes will likely need to be reverted after containment to restore normal operations within the environment, and a failure to document these changes can lead to wasted time and effort during recovery when it’s time to restore full operations.
5.3.2 Remove Power
In this approach, the responder removes power from impacted hosts by pulling the power cord. Removing power suddenly will halt any activity on the system immediately. The benefit of this action is that any files that are not encrypted or deleted will remain that way.
Tip: Shut Down Computers Cautiously
It is unwise to shut down an infected computer using a software or hardware shutdown button. A soft shutdown of this type can be undermined by the ransomware software itself or trigger additional destructive activities.
The downside is that a sudden power shutdown could potentially corrupt the entire filesystem. This is always a risk when suddenly shutting down power, but the risk is heightened due to the ransomware software’s behavior. For example, some variants of ransomware encrypt files in place. If power is removed while a file is being encrypted, then the files will likely be corrupted and may not be recoverable even with the help of a decryption tool.
Knowing the variant of ransomware involved can help you gauge the risk of corruption. For example, certain variants create an encrypted copy of files and then subsequently delete the original. In these cases, the risk of corruption is lower, because the original file still exists until the encryption process is complete.
Coming up in Section 5.7.3.2, we’ll discuss the order of volatility of evidence. For now, know that valuable information can be recovered from volatile evidence sources on affected hosts (e.g., the CPU cache, RAM, active network connections). This volatile evidence will be lost if power is removed from the system. It is often wise to quickly capture RAM from a system prior to shutting off the power, depending on the speed of the encryption process and the risk that the attack poses to the organization.
5.3.3 Kill the Malicious Processes
In any ransomware attack, some processes running on some computer in the environment are actively encrypting and/or deleting files. If responders choose not to cut the power, identifying the encryption process is key to halting the ongoing denial attack.
Identifying the ransomware encryption process can be tricky. For example, the Maze group was known to hollow out the legitimate svchost.exe process and inject encrypting software into it, so that the malware would appear to be a legitimate Windows process. The Dharma malware created a look-alike executable that would appear to be a Windows system utility, such as winhost.exe.
Encryption is a resource-intensive process. Responders can look for indicators of this activity such as high CPU usage or suspicious parent processes. In the case of Dharma, for example, if you examined a process tree, you could see that the process was spawned by a service that was not a Windows system utility. You can also identify these suspicious processes using a forensic memory analysis tool such as Volatility.
Responders can also track down the source of ransomware encryption by identifying the user associated with the encrypted files, analyzing the timeline, and in some cases leveraging this information to trace the attack back to a specific workstation or server.
Once you identify the process (or processes) that are encrypting data, kill them using a command prompt or Task Manager. Keep in mind that ransomware is often implemented using multiple processes, which can potentially run on different systems. Note also that killing processes that are hollowed-out Windows services can lead to system instability.
Tip: Use Trusted Tools
Often, the adversary modifies system tools such as Task Manager or command-line tools so that responders can’t see or effectively kill suspicious processes. You can prevent this by running your process analysis tools from an attached drive rather than relying on potentially suspect system tools. The Windows Sysinternals utilities are a useful toolkit for response purposes.
5.4 Disable Persistence Mechanisms
Adversaries often modify the victim’s systems to ensure that the malicious activities can continue, even if a process is killed or the system reboots. For example, depending on the specific variant used in the attack, ransomware may install additional malicious binaries throughout compromised hosts to create a persistent environment that will re-encrypt or destroy data periodically. In cases where ransomware is set to run automatically, the specific mechanisms need to be identified so malicious activities do not begin again during the recovery phase.
Identifying the adversary or ransomware strain can give responders a leg up when determining the best process response, as well as when evaluating the risk of ongoing encryption and persistence mechanisms. Although adversary tactics are constantly evolving, some behavioral characteristics are known to be associated with certain groups.
Adversaries typically create persistence using one or more of the following methods:
Secondary monitoring process
Scheduled tasks
Automatic startup
Many adversaries will establish local system persistence using methods like these. If a defender does not identify these persistent processes early, there is a chance that the entire network could fall victim again.
Heads Up! Evasion Tactics
Highly successful ransomware strains and RaaS products such as Maze or Ryuk may purposefully minimize system changes and/or purge their software from the system upon completion. This reduces the risk of persistence, but also inhibits forensic analysis of the malware and impedes an investigation. In addition, these adversaries often have privileged access to network administration tools and core servers, enabling the reinstallation of ransom-ware even if defenders clean a specific system.
5.4.1 Monitoring Process
Often, the adversary will include a secondary program alongside a malicious executable designed to monitor the status of the process. If a primary malicious process stops, the monitoring program can detect this fact and restart it. The monitoring process (also referred to as a watchdog timer) can be more difficult to identify, since it does not require extensive use of the CPU or other system resources and can be made to resemble a legitimate system service.
5.4.2 Scheduled Tasks
The adversary may set up scheduled tasks in the operating system to relaunch ransomware-related executables periodically. For example, on Windows systems, look for AutoRun keys in the registry.
First, responders should delete any scheduled tasks that invoke ransomware or another malicious process. If you need to take quick action, consider removing all scheduled tasks. While this may impact normal system performance depending on the victim’s custom configuration, typically Windows can operate without scheduled tasks. Make sure to preserve evidence (such as registry keys) before modifying system configuration.
5.4.3 Automatic Startup
Many adversaries modify the system configuration so that if an infected computer is rebooted, the malicious process will relaunch. This is especially damaging with ransomware strains such as Dharma, as the new process will utilize a new encryption key following the reboot. As a result, an infected system may end up having files encrypted with two or more keys, depending on the number of times the infected host was rebooted.
Case Study: Double (and Triple, and Quadruple) Encryption
A midsized health care organization was infected with the GandCrab ransomware in 2018. Several workstations were infected, and all of them had access to the organization’s central file-sharing repository.
Unfortunately, this meant that the GandCrab encryption software ran multiple times on the same files. The first infected workstation encrypted all files in the shared repository and dropped the ransom note containing the link to the custom portal required to purchase the matching decryptor. Without this link, it was impossible to get the decryptor.The second infected computer subsequently encrypted all of the files again, including the ransom note. Then it dropped a new ransom note of its own. In consequence, to fully decrypt the files, the victim had to purchase the decryptor using the note left by the second infected computer, so that it could recover the first ransom note and gain access to purchase the original decryptor. Whew!
By the end of the incident, four separate infected computers had encrypted the files, and responders had to go through the process of purchasing and decrypting four nested layers of encryption.
5.5 Halting Data Exfiltration
Stopping any current data exfiltration activity is also a top priority. If you are investigating a cyber extortion case and you are not sure whether data is being actively exfiltrated, assume that it is. At the time of this writing, approximately 84% of all ransomware attacks involve a threat to leak stolen data.2
2. “Law Enforcement Pressure Forces Ransomware Groups to Refine Tactics in Q4 2021,” Coveware (blog), February 2, 2022, www.coveware.com/blog/2022/2/2/law-enforcement-pressure-forces-ransomware-groups-to-refine-tactics-in-q4-2021#:~:text=84%25%20of%20Ransomware%20Attacks%20Included%20Data%20Exfiltration&text=RaaS%20affiliates%20expect%20exfiltrated%20data,pay%20a%20cyber%20extortion%20demand.
The adversary may steal data from a local network, cloud repository, mobile device, or any other repository. As a result, responders must consider all data repository locations when attempting to halt exfiltration.
Immediately check alerts, logs, and outbound network traffic for any signs of suspicious outbound communication. If any are identified, stop data exfiltration as quickly as possible while balancing operational needs. Here are some tactics commonly used to stop data exfiltration:
Block suspicious outbound network traffic at the perimeter firewall, or an intermediary internal firewall if available.
Block access to any cloud services or file-sharing sites used by the adversary to transfer data, such as Dropbox or MEGA.
Disallow the use of utilities such as FTP applications, PowerShell, or Win-SCP if they’re not necessary.
Restrict data repository access by modifying permissions, roles, and application configurations as appropriate.
Remove any email forwarding rules that were created by an adversary.
Take other steps to block data exfiltration as appropriate.
Victims might consider cutting off all network traffic as a temporary measure. This decision needs to be made on a case-by-case basis and consider the victim’s business model, as well as weigh the potential damage of network cutoff against the potential benefits. Cutting off all network access is the “nuclear option” for most organizations, but nothing is really off the table after a cyber extortion attack is discovered.
5.6 Resolve Denial-of-Service Attacks
File encryption and exfiltration are not the only types of “denial” cyberattacks. Access to network resources and functionality can also be severely impacted by a distributed denial-of-service (DDoS) attack. In a DDoS attack, an adversary intentionally overloads the online infrastructure with network traffic and stops the services from operating normally. These types of attacks can be crippling to network infrastructure and require immediate attention.
Over the years, cybercriminals have attempted to leverage DDoS attacks in a variety of cyber extortion cases. For example, in August 2020, a group known as “Lazarus Bear Armada” launched a series of carefully planned DDoS attacks against targets in the financial and travel industries.3 Depending on the resilience of the specific target and its position in the technology supply chain, a DDoS attack can either impact a single organization or cause widespread ripple effects.
3. Arbor Networks, “Lazarus Bear Armada DDoS Extortion Campaign—December 2020,” NetScout, www.netscout.com/blog/asert/lazarus-bear-armada-ddos-extortion-campaign-december-2020.
In a DDoS extortion attack, criminals launch a DDoS campaign against a victim, and then offer to end the attack in exchange for a ransom payment. For example, in January 2021, security company Radware warned that customers had been receiving extortion emails threatening damaging DDoS attacks unless they paid the adversaries 5 to 10 Bitcoins (worth approximately $150,000 to $300,000 at the time).4
4. Catalin Cimpanu, “As Bitcoin Price Surges, DDoS Extortion Gangs Return in Force,” ZDNet, January 22, 2021, www.zdnet.com/article/as-bitcoin-price-surges-ddos-extortion-gangs-return-in-force/.
Akamai, an Internet security company, warned shortly thereafter that DDoS attacks in 2021 had become “more targeted and much more persistent.” In some cases, adversaries leveraged sophisticated attack strategies, “rotat[ing] through multiple DDoS attack vectors trying to increase the likelihood of disrupting the back-end environments.”5
5. Ionut Ilascu, “800Gbps DDoS Extortion Attack Hits Gambling Company,” Bleeping Computer, March 31, 2021, www.bleepingcomputer.com/news/security/800gbps-ddos-extortion-attack-hits-gambling-company/.
To defend against a DDoS attack, consider leveraging a DDoS mitigation service. Third-party providers such as CloudFlare, Akamai, and others specifically offer DDoS mitigation options, which include artificial intelligence (AI)–driven threat identification and response, distributed caching proxies, and more.
5.7 Lock Out the Hackers
Adversaries often remain in the compromised environment indefinitely, monitoring communications and potentially sharing access with others, unless responders deliberately and completely eradicate them.
During the initial steps to contain an incident, responders typically do not have a complete understanding of the adversary’s method of entry, or even the current access mechanisms employed. This is normal. Despite this obvious challenge, it is absolutely critical to lock out the adversary from the system, so as to ensure that the damage is contained and additional malware is not installed during the course of the investigation.
Tactics for locking out the hackers include the following:
Kill remote connection services
Implement password resets
Roll out multifactor authentication
Restrict perimeter communications
Review and minimize third-party access
Mitigate risks of compromised software
We discuss each of these methods in the following subsections.
5.7.1 Remote Connection Services
Many enterprise environments allow employees, contractors, vendors, IT staff, and many others to have remote access to their networks, often in the form of Remote Desktop Protocol (RDP), virtual private networks (VPNs), and Secure Shell (SSH) logins. It should come as no surprise that remote access systems are also a prime target for would-be attackers. The access is already configured, and in many cases the only thing an adversary needs to exploit these systems is a username and a password.
During the initial response to a cyber extortion incident, responders should consider taking the following actions to lock down remote connection services:
Disable any port forwarding that allows RDP from the public Internet.
Shut down any VPN access points.
Disable SSH services from the public Internet.
Kill any and all active remote sessions for RDP, VPN, SSH, and other remote connection tools, unless they are absolutely necessary and have been vetted by the response team.
5.7.2 Reset Passwords for Local and Cloud Accounts
Password resets are an essential tactic for quickly reducing the risk of ongoing damage, and should be implemented even before the scope of a compromise is fully understood.
5.7.2.1 Which Passwords Should You Reset?
During the first response, it is usually advisable to conduct a mass password reset for all domain-connected accounts, as well as cloud services such as Microsoft 365 and QuickBooks. This will prevent the adversary from accessing services using stolen passwords, and it often reduces the risk that the adversary will reenter the organization using sanctioned remote access tools.
Every environment is unique. When determining whether to reset passwords, or establishing a timeline for doing so, responders should weigh the potential benefits of a password reset with the difficulty and cost.
5.7.2.2 When and How to Reset Passwords
Simply changing passwords without preparation can be both ineffective and potentially disruptive. If an adversary has already established a persistent foothold in your network that does not rely on privileged access (such as a RAT), then changing passwords may just result in the new passwords being quickly exfiltrated as well.
Local system password resets should be performed as soon as possible after a host is determined to be no longer compromised. It is safest to perform this operation on an isolated network segment, separate from the compromised environment. Responders should be prepared to repeat this process as needed if an infected computer is connected to the network of “clean” computers.
Additionally, responders should make sure to reset Kerberos access tokens, as a means to prevent exploitation using token-based attacks against the domain. Refer to vendor documentation for the full reset process.
Cloud services are also a prime target for exploitation and need to be addressed quickly. Responders should revoke all active sessions to their cloud environments and then reset the passwords for all accounts. Passwords for cloud services should be reset only using a computer that is not infected by malicious software and is not part of the infected network.
5.7.2.3 New Password Selection
In all cases, new passwords should be secure, strong, and unique for each account.
It is not uncommon to encounter a shared local administrator password within an enterprise network that IT staff use for initial setup and continued maintenance. In such a case, responders should consider configuring a local administrator password tracking solution like Local Administrator Password Solution (LAPS) to avoid sharing passwords between multiple devices.
5.7.3 Audit Accounts
Many adversaries routinely alter account permissions or create new user accounts. Such changes can involve modifying file share access, software installation permissions, remote access, and more. If responders fail to identify and correct these changes, it can leave the door open for the adversary to regain entry to the environment and disrupt or negate recovery efforts.
Responders should audit and evaluate user accounts while looking for suspicious signs such as the following:
Recently created accounts
Newly enabled remote access permissions
Administrative permission assignments
New file share access
Password changes
Make sure to immediately disable or restrict any accounts found to have been altered or created by an adversary. Access can be reestablished for authorized users once their accounts have been verified and properly secured.
5.7.4 Multifactor Authentication
Determine whether multifactor authentication (MFA) is set up for all remote access to the victim’s technology environment. If it is not, or if it is only partially set up, identify gaps. Strongly consider rolling out multifactor authentication for all remote access, including VPN, webmail, and cloud environments.
While lack of MFA might not seem to be directly related to the cause of a cyber extortion attack, a speedy rollout of MFA can often dramatically reduce the risk of an ongoing compromise or reinfection due to password theft.
Many organizations spend months planning for a potential rollout of MFA, then miraculously implement it completely over the span of a weekend after a cyber extortion incident takes place. Although this accelerated time frame is not ideal, it is often a wise step for reducing risk.
5.7.5 Restrict Perimeter Communications
Restricting communication, both incoming and outgoing, may be the quickest route to locking out any ongoing unauthorized access. However, be careful: Perimeter network restrictions can also become a roadblock if responders need to access critical portions of the organization’s infrastructure remotely. If possible, restrict the following services:
RDP services, even if they’re running through alternative ports
VPN access, unless the access is from a specifically allowlisted IP address
SMB, FTP, or any other file-sharing traffic
SSH access
Any nonessential services
5.7.6 Minimize Third-Party Access
Third parties, including managed service providers (MSPs), technology vendors, software providers, and others, can unintentionally act as a vector for unauthorized remote access to customer network. That outcome is particularly likely if their own technology ecosystem is compromised first.
For example, it is common to see cyber extortionists compromise MSP accounts with administrative privileges and leverage these to launch extortion attacks against the MSP’s customers. In the event of a cyber extortion attack, it’s typically a good idea to immediately notify the victim’s MSP. This way, the MSP can provide support, and also adapt in response to any remote access or network configuration changes. In addition, any administrative accounts that an MSP uses to administer an environment should be quickly restricted or disabled, along with remote access software. This way, the adversary cannot easily leverage the MSP’s privileged access if it has been compromised.
Technology vendors pose similar risks, particularly if they have ongoing remote access to the victim’s environment. While specific vendors may not have full administrative accounts across the entire technology ecosystem, they may have access to more than a standard user, making them a target for exploitation.
To reduce the risk of an attack:
Third-party remote access to the network should be minimized during the early phases of the response.
All vendor accounts should be disabled unless absolutely necessary.
Consider disconnecting any hardware or software managed by a potentially compromised vendor.
5.7.7 Mitigate Risks of Compromised Software
Unfortunately, software products may be vectors for injection of malicious backdoors or other malware into the victim’s environment, as discussed in Section 1.6. These situations can be particularly knotty and difficult to resolve, especially in cases like the SolarWinds and Kaseya incidents, in which the affected software was integral to the victims’ technology environment.
In the early phases of the response, it is wise to minimize software running in the environment until the source and scope of the compromise are fully understood. If evidence suggests that the adversary exploited existing software or leveraged an implanted backdoor:
Contact the vendor right away to notify and obtain guidance.
If possible, remove the affected software from the network.
Otherwise:
Take steps to eradicate the adversary.
Install software patches that address the issue as soon as practical.
If patches are not yet available to fully address the issue, refer to government and vendor advisories for risk mitigation steps.
5.8 Hunt for Threats
Threat hunting refers to the process of proactively and manually searching a technology environment for indications of threats. Threat hunting often begins immediately upon detection and is used to identify suspicious activity within the environment so that the adversary can be fully eradicated.
Today, threat hunting is an integral part of an effective response to cyber extortion. To be effective, this activity is typically conducted by an experienced professional using specialized tools, and aims to detect subtle threats that automated software such as antivirus suites may miss. An experienced threat hunter can cut the period of active network compromise by days, if not weeks, if they are provided with quick access to a compromised network and responders are prepared to actively leverage their results. Threat hunting typically continues long after containment to reduce the risk of reinfection and prevent future incidents.
A comprehensive approach to threat identification is essential for effective eradication. In this section, we provide a high-level overview of threat hunting methodology, techniques, sources of evidence, tools, and staffing.
5.8.1 Methodology
Threat hunting is a cyclical, loosely defined process involving the following phases:
Collect information
Create and refine the hypothesis
Hunt
Identify threats
In each phase, the threat hunter leverages the results from the previous phase to hone in on suspicious activity and identify novel or advanced persistent threats.
Threat hunting is an ongoing process that needs to be included in each phase of the response, and will likely continue after recovery is complete. Organizations should strongly consider adding proactive threat hunting as a part of their cybersecurity testing routine, since adversaries are constantly innovating and finding new ways to gain persistence.
Case Study: Hunting Trip
A private charter school in the U.S. Northwest was hit with the Conti ransomware and completely locked up. The ransomware gang had stolen hundreds of student records, as well as other files, and threatened the school with double extortion. Local IT staff used Windows Defender to scan the network after it was encrypted with ransomware. Windows Defender did not identify any signs of additional malware.
The authors of this book were hired to conduct triage and ensure that the threat was fully eradicated. Knowing that the Conti ransomware is typically deployed using a remote access tool (RAT), we deployed an EDR threat hunting tool. It didn’t take long to find the Trojan (QBot), a powerful threat distributor and information stealer, still lurking on the network. Once we identified the RAT, we worked with the local IT staff to fully eradicate the underlying threat and verify that the network was free of malware.
Had the school not invested the time and resources to carefully monitor the network and conduct threat hunting, a dangerous RAT would have remained in the network, and would likely have led to reinfection.
5.8.2 Sources of Evidence for Threat Hunting
Here are the minimum sources of evidence needed to conduct effective threat hunting on an infected network:
Network flow records and/or firewall logs (both internal and external)
Event log data
Process activity
User activity
Authentication events (successful and failed)
Other information streams can supplement this data and can inform the threat hunting process.
5.8.3 Tools and Techniques
Threat hunting is not dependent on one single piece of software. Instead, a trained responder needs to understand how a network functions normally, then leverage tools as appropriate to identify behavior that is outside of the normal parameters.
During the entry and expansion phases of the attack, the adversary may have deployed tools to establish persistence, or to cause reinfection if the malware is eradicated using normal methods. These artifacts need to be rooted out to minimize the risk of future compromise.
Commonly, EDR software is used as the primary threat hunting tool. Possible tools include CrowdStrike,6 Carbon Black,7 SentinelOne,8 and other EDR toolkits. Responders can also leverage the free and open-source ELK stack, which is composed of Elasticsearch, Logstash, and Kibana. Often, the best threat hunting software is the tool that is already installed in the victim’s environment; every moment is precious during a cyber extortion response, and “living off the land” can be the fastest means of eradicating the threat if the existing tools are sufficient.
6. CrowdStrike, www.crowdstrike.com/.
7. VMware Carbon Black, www.carbonblack.com/.
8. SentinelOne, www.sentinelone.com/.
5.8.4 Staffing
Human involvement is of the utmost importance during threat hunting because threats evolve far more quickly than defenders can develop and deploy programmatic software.
For example, consider an adversary who has gained remote access to an IT administrator workstation within a network. The adversary uses stored passwords on the workstation to log in to a central database server, while pivoting through the workstation. This all-too-common scenario may not trigger an alert from the database server because the login characteristics are effectively “normal.” Moreover, adversaries may use a variety of tactics to blend in with normal remote access tools or network behavior and evade detection.
This is where a manual hunt becomes central to the response strategy. A professional threat hunter may well detect unusual activity based on context and experience, while automated tools remain blind to the threat.
Most organizations do not have a full-time threat hunting specialist on staff, ready to be deployed at a moment’s notice when a cyber incident occurs. Even for those that do, there is a benefit to hiring a threat hunting consultant who routinely works with many organizations. During a cyber extortion crisis, outsourcing can lighten the load of a victim’s already taxed internal staff, and enable the victim to tap into specialized expertise.
5.8.5 Results
Since threat hunting is a cyclical process, it’s important to analyze and communicate results at regular intervals. Along the way, the hunt may turn up anomalous activities that are not normal for the environment, such as the following outliers:
Unusual external network connections
Increases in inbound or outbound network traffic
Abnormal file or device access
Suspicious process invocations
Multiple consecutive failed authentication events
Activity at odd hours
Once such anomalous behaviors are identified, they need to be documented, vetted, and used to further refine the hypothesis.
Ultimately, when a verified threat is discovered, information about it needs to be quickly and clearly communicated to responders, who in turn can quickly eradicate it. Eradication can take many forms, but often includes the following steps:
Removing suspicious hosts or virtual machines (VMs) from the environment
Deactivating unexplained or malicious user accounts
Disabling newly installed or suspicious software applications
Eradicating any other sources of potential threats
As threats are identified and verified, make sure to generate signature data and update security solutions to leverage new information.
Heads Up! No-Malware Attacks
According to Crowdstrike’s 2022 Global Threat Report, 62% of malicious activity detections in Q4 of 2021 were malware-free. Increasingly, hackers are co-opting normal IT administration tools and using them to push out malware or move laterally throughout an environment. As a result, antivirus tools flag legitimate pieces of software as potentially malicious with increasing regularity. This includes file transfer software such as FTP applications, penetration testing tools such as Cobalt Strike, and commonly used utilities such as PSExec. Exercise caution and verify that programs are actually malicious before terminating them.
5.9 Taking Stock
As the emergency activities slow, responders should conduct a “pulse check” (see Chapter 4) to assess the outcome of the triage activities and the victim’s operational state. This assessment should include the following items:
Effectiveness of the containment activities:
– Was file encryption halted?
– Was data exposure halted?
– Has the denial-of-service attack been contained?
Risk of continued adversary access to the victim’s environment
Volume and criticality of data that is currently unavailable or at risk of exposure
Services or systems that are currently unavailable
Current status of business operations
Any additional resources that are needed
Based on the results, review and update your response strategy, which will continue to evolve over time.
5.10 Conclusion
Containment is a critical part of the early response process. In this chapter, we presented methods frequently employed in containment activities, including tactics for quickly stopping file encryption/deletion, halting data exfiltration, resolving denial-of-service attacks, and locking out the adversary.
During containment, response teams face special challenges due to their lack of information in this phase, combined with the high potential impact of their decisions. It is critical for responders to maintain good communications and have clear decision-making processes in place to achieve the best possible outcomes.
In the next chapter, we will discuss methods for investigating a cyber extortion crisis that will provide the organization with actionable intelligence and inform further response efforts.
5.11 Your Turn!
Every cyber extortion incident is unique. The response team’s options and priorities will vary depending on the victim organization’s industry, size, and location, as well as the details of the incident itself.
Based on what you learned in this chapter, let’s think through key elements of containment.
Step 1: Build Your Victim
Choose one characteristic from each of the three columns to describe your victim’s organization:
Industry | Size | Location |
|---|---|---|
Hospital | Large | Global |
Financial institution | Midsized | United States |
Manufacturer | Small | European Union |
Law firm |
| Australia |
University |
| India |
Cloud service provider | Country/location of your choice | |
Organization of your choice |
|
|
Step 2: Choose Your Incident Scenario
Select from one of the following incident scenarios:
A | Ransomware strikes! All of the victim’s files have been locked up, including central data repositories, servers, and workstations. |
B | A well-known cyber extortion gang claims to have stolen all of the victim’s most sensitive data and threatens to release it unless the victim pays a very large ransom demand. The gang posts the victim’s name on their dark web leaks site, along with samples of supposedly stolen data. |
C | Double extortion! Both A and B occur at the same time. |
D | The victim is hit with a denial-of-service attack on its Internet-facing infrastructure that slows its access and services to a crawl. The adversary threatens to continue and even escalate the attack unless a ransom is paid. |
Step 3: Discussion Time
Your victim must contain their cybersecurity incident. Given what you know of the victim and the scenario, answer the following questions:
Based on the scenario you chose for your victim, list three appropriate goals for containment.
What is the worst-case scenario if the incident is not contained?
What are some strategies that the incident response team can use to stay in communication during containment? Consider how normal communications may be impacted by the incident.
There is evidence that the adversary had access to the infrastructure, and the incident response team is not sure if the adversary is still actively accessing it. Name three steps that responders can take to lock out the adversary.
Given the type of attack, would you recommend resetting passwords network-wide? Why or why not?