CHAPTER 22
Incident Handling
Incident handling is the first step in an actual recovery process. The activity is undertaken by the organization to manage the consequences of a breach to minimize both tangible and intangible damage. Examples of breaches include intrusion, cybertheft, and denial-of-service and virus attacks. Incident handling is part of information assurance, but it is a reactive control. Some organizations define incident handling as “information security incident handling” or “security incident handling.” In all contexts, the functions remain the same regardless of the name.
Incident handling is included sometimes as part of an organization’s business continuity plan because it provides the approach to respond quickly and efficiently to unexpected events.
The best way to act on an incident and minimize the chances of a mistake is by establishing predetermined procedures. The procedures should be clear and well documented to ensure that after an incident, the proper steps are followed. Clarity is extremely important to reduce the risk of overlooking actions required by law. An incident-handling plan should be compliant with the applicable laws and regulations. Refer to local civil and criminal laws for guidance. Here are some sample items you would include in your research:
• Evidence management
• Criminal investigations
• Civil investigations
• Administrative investigations
• Response to security incidents
• E-discovery
• Intellectual property
This chapter is an overview of the importance of incident-handling capabilities for an organization and points out the best practices in implementing the incident-handling process. You should to refer to Chapter 23 for details on computer forensics, which is a component of incident handling.
Importance of Incident Handling
Regardless of the type of organization or size, they all face the risk of being affected by an incident. Incidents may completely disable an organization’s operations if not handled properly. Unfortunately, some organizations handle incidents by ignoring them. It is only a matter of time before they realize what ignoring incidents can do to their business. Organizations must react appropriately to incidents not only to limit damage but also to ensure they learn and update their operations to prevent further incidents.
Handling an incident is straightforward if the incident-handling team is well prepared. Even when caught off-guard, planning and preparing for incident handling allows the information assurance team to identify and recover the system and prevent further damage. In conjunction with computer forensics and evidence management, incident handling ensures that digital evidence collected is admissible, authentic, complete, and reliable for legal action (refer also to Chapter 23). Initially, all incidents should be handled as though they have criminal or civil litigation implications. Once an incident has been determined to be criminal or subject to litigation, earlier mistakes or oversight may make information obtained by the information assurance team inadmissible to the courts.
A well-handled incident is good for morale since it allows IT personnel to identify weaknesses, take precautions such as update patches, and sharpen their skills. Therefore, the information assurance posture of the organization improves. Further, during the actual incident-handling process, data collected about threats and vulnerabilities will be useful to the risk analysis process (see also Chapter 11). The data can be analyzed, correlated, and grouped to analyze hackers’ attack patterns. Statistics on different types of incidents in the organization may be used to prepare reports about an organization’s information assurance posture.
Incident handling tests the communication and teamwork capabilities of an organization. Internally, management must be aware and organized to receive incident updates. If properly communicated, feedback about the incident helps system administrators and system owners improve their systems. Communication between functional groups such as public relations and legal can be strengthened through the development of mature incident response practices. Communication with external parties such as Internet service providers (ISP), vendors, law enforcement agencies, and the press can be developed and refined through the incident response process. When organizations establish working contacts with law enforcement, vendors, and ISP partners, it provides a significant time advantage for future incident response actions.
Incident Reporting
Incident handling revolves around two words: incident and event. An incident refers to actions that result in damage and form substantial threats to the information system. Except for incidents caused by unforeseen natural disasters, IT incidents can be largely avoided by an organization. Events are large or small abnormalities detected in the network or system.
You can prove an event has occurred, there is always evidence or we wouldn’t be concerned. The event can be a strange message flash on the screen or even a sound. Something that appears in the audit trail or log files may also be an event. Recognizing and recording these events is important. Events become incidents when it is clear measurable damage may occur or an eminent threat exists. Security operations centers and organizational SIEMs are often flooded with events. Some of the larger SIEMs see more than 500,000 events per hour; however, only a few become incidents. One of the most difficult tasks an organization faces is detecting incidents in a large collection of events. Even so, it is important for an organization to gather reliable evidence of incidents since laws are in place to protect organization. These records may be used in a later court case. Be aware that attackers sometimes use tools to alter or delete traces of their activities in log files.
The evidence is more convincing if organizations produce contemporaneous, accurate sources, and records for information. An organization with a timely, accurate, and thorough incident response policy, operational procedures, and standards is far more likely to succeed in prosecution, litigation and recovery. Organizations execute incident response procedures in the way they practice them. Incident response practice and testing should be part of every organization’s continuous monitoring approach and risk management strategy.
An important question to ask is, how and when is an event reported? Accurate, thorough, and timely information by witnesses contributes greatly to the success of an incident-handling process. Communicate all events and weaknesses through a robust pre-established event reporting and escalation procedure. Document the procedure in the organization’s information assurance policy.
All employees, vendors, and third-party users should know the incident response procedures and should be encouraged to report an event as soon as possible to prevent an escalation to an incident. They should know the channels and procedures for reporting various events and the effect of incidents on a company’s assets.
Organizations must establish proper reporting channels such as the use of e-mail, telephone, and reporting forms (remember, an incident may disable all electronic media). Management should use awareness and training to ensure that everyone knows the appropriate channels. Additionally, rules of behavior and contractual agreements should mandate the reporting of events. Everyone must report all identified and suspected information assurance weaknesses in accordance with the event reporting and escalation procedure.
In the United States, the U.S. Computer Emergency Readiness Team (CERT) is part of the United States Department of Homeland Security (DHS) and provides incident response support and incident analysis. There are reporting procedures and reporting templates provided, which organizations may find useful.
Generally, an organizational incident reporting procedure should encompass the following:
• Mechanism for publishing the identified incidents
• Mechanism for reporting and recording information security incidents
• Disciplinary process for those who commit an information security breach or willingly refuse to report a breach
• Relevant instructions such as the need to record incidents promptly
Another example is the Danish Centre for Cyber Security (CFCS), which analyzes security incidents, proposes solutions, and warns others of similar incidents while assuring anonymity and confidentiality of all parties. It also coordinates information for foreign response teams and police. The Danish DKCERT is strictly advisory for civilian organization but has authority through its MILCERT component to order changes to the Danish Department of Defense’s military IT and communications systems. To support both operations and AT&E, CFCS publishes articles, alerts, advisories, and news regarding vulnerabilities and precautionary measures.
Incident Handling Process
Incident handling centers on preparation; if the organization is not prepared, the process fails. It requires flexibility to manage the processes, procedures, and countermeasures. Flexibility allows the organization to react to different attacks and environments. The process suggested in this illustration is a synthesis of best practices in the industry.
Figure 22-1 shows the process has phases centered on preparation identification, containment, eradication, recovery, and review. The linkage among the phases is a compass or a road map for incident handlers within an organization. The first phase of the process, identification, is inextricably tied to preparation. Together, they are day-to-day practices for incident handlers. Before taking any action about an incident, handlers need to be prepared to identify events that may grow into incidents.
Figure 22-1 Incident handling process
Phase 1: Preparation
Preparation and planning are essential to address any incident effectively. Preparation is a team effort that not only includes internal resources but also those of vendors, contractors, law enforcement, legal counsel, and perhaps national or regional CERT. Budget the most time for preparation and requires the most effort in the preparation process. Preparation is continuous process. Skills and alertness fade over time without AT&E and fresh exercises. Preparation includes defining the incident-handling policy, establishing logging standards and warning banners, setting up the infrastructure and resources, installing the necessary environmental controls, and documenting procedures and baselines.
Every organization should establish an incident-handling policy and warning banners if local laws allow. For example, in the United States, a good practice would be the use of warning banners to inform individuals connecting to the system that access to the system is monitored and recorded to ensure systems are used for authorized purpose only. Unauthorized access, use, or modification is prohibited, and unauthorized users may face charges and criminal or civil penalties if allowed by local law. The policy and banner help define an incident. When legally permissible, users must be required to acknowledge and accept the banner before using an organization’s information systems. Other economies may have other requirements or customs.
An incident-handling approach establishes the conditions and responses for an incident. The incident response policy establishes and authorizes the approach to all incident handling organization-wide and requires approval from senior management to ensure a mandate for smooth execution and stimulates employee buy-in.
Once the policy and approach are refined, form the incident-handling team. Draw the members of the team from different backgrounds and discipline such as computer and physical security, system administration, network management, legal, human resources, risk management, and corporate communications. In the event of a serious incident, the technically focused team members will perform onsite investigation, evaluate the situation, and make recommendations. As noted in Chapter 5, incident-handling team leaders should hold professional certifications and credentials validating their expertise, relevant experience, and commitment to continuing professional education. Credentials and certifications applicable to incident handers include, but are not limited to the following:
• International Information Systems Security Certification Consortium (ISC)2
• Certified Information Systems Security Professional (CISSP)
• Information Systems Security Architecture Professional (CISSP-ISSAP)
• Information Systems Security Engineering Professional (CISSP-ISSEP)
• Information Systems Security Management Professional (CISSP-ISSMP)
• Certified Authorization Professional (CAP)
• Certified Cyber Forensics Professional (CCFP)
• Certified Secure Software Lifecycle Professional (CSSLP)
• HealthCare Information Security and Privacy Practitioner (HCISPP)
• SSCP (Systems Security Certified Practitioner (SSCP)
• SysAdmin, Audit, Network, Security (SANS)
• Certified Incident Handler (GCIH)
• Certified Intrusion Analyst (GCIA)
• Certified Forensic Analyst (GCFA)
• Certified Forensics Examiner (GCFE)
• Information Systems Audit and Control Association (ISACA)
• Certified Information Systems Auditor (CISA)
• Certified Information Security Manager (CISM)
• Certified in Risk and Information Systems Control (CRISK)
• EC-Council
• Certified Ethical Hacker (CEH)
• Computer Hacking Forensic Investigator (CHFI)
• EC-Council Certified Incident Handler (ECIH)
Incident handling requires an emergency/crisis communication plan. This plan is part of business continuity planning (BCP) and should contain a call list for escalation purposes based on severity or criticality of an incident and a call tree to allow all employees to be contacted. This emergency communication plan should be distributed to all personnel in charge and affected users.
Organizations should establish a primary contact point and develop a list of objectives for the incident-handling team. All reporting facilities and mechanisms should be in place to allow employees to report an incident.
Organizations must conduct AT&E exercises to ensure an incident-handling team is ready. The training should be practical rather than theoretical. It should involve rehearsal of potential incident scenarios, use of techniques and tools, forensic evidence handling procedures, administration of various system and hardware, or even games to simulate an incident and the associated activities. “Capture the flag” and “Defense on a Cyber Defense Simulator (CDS) or CyberRange” events offer excellent venues for incident-handling teams to hone their skills.
Phase 2: Detection/Identification
In incident handling, time is crucial. Organizations must take immediate and correct actions after identifying an event or incident. Even if it is later discovered that the event was a false alarm, the incident-handling team gets practice on responding to an event or incident. To identify events or incidents accurately, team members must be aware of the situation, be updated on the status, and be able to rapidly and accurately integrate the information provided.
During planning, designate an individual as the primary incident handler for the identification phase. The primary incident handler updates the management about the problem within a specific time. Organizations should also determine a line of succession should the primary incident handler be unavailable.
During identification, the incident-handling team performs an initial assessment of the validity of an incident. The team assesses the evidence in detail since a single event may be part of an attack chain or lead to litigation.
An incident can be detected by witnesses or through continuous monitoring sensor platforms such as firewalls, audit logs, or intrusion detection systems. Witnesses or systems notify the incident-handling team about suspicious events. Examples are unsuccessful login attempts, unexplained new user accounts, poor system performance, and denial of services.
Some incidents can be detected at the system level. For example, the network perimeter, a network-based intrusion system, firewalls, or routers detect suspicious events. For the host perimeter, the identification is usually detected by a personal firewall, anti-malware system, or IDS. Additionally, file integrity checkers and antivirus software are often monitored for suspicious behavior. Users as witnesses may also detect system-level perimeter intrusions by reporting odd behaviors.
While identifying an incident, the incident-handling team must establish a chain of custody to ensure that all relevant information and evidence is well documented, preserved, and protected. The incident may eventually turn out to be just an event or something that won’t be prosecuted; the incident response team can’t go back and put a chain of custody on events after the fact. If they do, they may compromise evidence. Incident-handling teams must always remember in many criminal courts the rule is reasonable doubt, and, therefore, their actions must be beyond reproach.
Phase 3: Containment
Containment is an immediate act. The organization’s incident response plan should focus on mitigating or stopping the damage. As the term implies, the objective of containment is to prevent the incident or situation from escalating. Where possible, an event should not be allowed to become an incident. If that fails, incident escalation and containment reaction may involve a change of passwords for affected systems, blocking of suspicious IPs, or the application of patches to connected systems. To prevent the spread of the problem, take appropriate actions during the containment phase.
As noted in Chapter 18, a structured review is always required before changing a configuration or system. A change made in error or containing errors is almost certain to exacerbate the problem. At any time, members of the incident-handling team must act in accordance with documented standard policies and procedures; otherwise, the integrity of the evidence will be questioned in court later. In addition, failure to use documented policies and practices may cause the incident-handling team to damage the organization’s systems further. Backups and validated baseline configurations for the original system must be in place for both restoration and forensic purposes. The incident-handling team must always preserve the original copy as evidence and ensure that it remains unchanged. Specific information regarding forensics is covered in Chapter 23.
When dealing with external attacks such as a denial of service (DOS), coordinate closely with the Internet service provider. As noted prior, establishing verified contacts for the ISP with authority to aid the organization is a task that must be performed prior to an event to be most effective. An ISP can provide assistance, especially when there is a large flow of packets flooding an organization’s network and services. Most ISPs are experienced in handling network-based exploits and denial of services. Some charge additional fees for services related to DOS attack mitigation. Organizations should decide through a risk assessment if additional protection is worth the price. See Chapter 11 for a discussion of the risk assessment process.
The incident-handling team performs an initial analysis during the containment phase by analyzing a backup copy of the system. The team looks for deviations from the known baseline. If an organization does not have a mature configuration management process as part of its change management process, this task becomes much more difficult. Differences between the baseline configuration and the system under analysis are documented and verified by the incident-handling team. These findings will begin the road map for containment and mitigation.
Organizations may choose to avoid alerting the intruder unnecessarily, by not conducting tracing and mitigation activities or beginning mitigation immediately. This delay decreases the probability of losing a trail to the attacker. If the event impacts mission-critical systems, the organization may have no choice but to begin containment and mitigation. If the organization can withstand the impact of the incident and would gain from understanding more about the attacker, they may want to continue monitoring the attack in a contained environment. Organizations must be cautious when they suspect criminal laws may have been broken. In many countries, individuals are required to report the commission of crimes such as child pornography to authorities immediately upon discovery. Organizations must work with legal counsel to determine what types of events or incidents require immediate law enforcement involvement. Regardless of the path chosen, the organization should then report the status of the analysis (both initial and ongoing) to the senior risk function of the organization. The CISO, CIO, CFO, COO, CEO, or head of the organization is most often briefed about the incident status, depending on severity and area of risk.
After the analysis, the incident-handling team determines the risk of continuing operations. The team should gather logs and diagnostic details from network devices, servers, SEIMs, and end points to determine the severity of the incident and suitable recovery approaches. The team should develop a written recommendation at this point. The recommendation should contain options with associated impacts such as whether to shut down the affected machine or not. These options should be reported to the business or mission owner. If an organization has a C&A process in place, the accreditation official should be updated as well. The CISO, CIO, CFO, COO, CEO, or head of the organization is most often briefed about the incident status and chosen remediation actions, depending on severity and area of risk. In severe incidents, the organization may choose to fail-over to an alternate processing or storage site in accordance with its business continuity plan.
Phase 4: Eradication
Eradication is the most difficult phase for incidents involving malware because malicious codes need to be removed completely and safely. In addition, resolving this phase depends on the severity of an incident and the experience of the incident-handling team.
The eradication phase includes identification and elimination of vulnerabilities that have been exploited. Ideally, the incident-handling team should isolate a system or network to determine how the attack began and, if appropriate, initiate forensic analysis (Chapter 23).
To ensure the attacker does not perform the same attack on other machines within the network, it is wise to perform a vulnerability scan on the network and other systems to identify related vulnerabilities. Additionally, all IDS or IPS systems, host or network based, should be updated with signatures developed from the vulnerability or attack. Additional systems should be prioritized based on value and risk for cleaning, eradication, and re-securing.
These are steps to think about during the eradication phase:
• Even if the incident-handling team was able to successfully stop the malware from executing stubborn forms might reappear the next time a system is started. Where is the problem software located?
• Has the incident-handling team examined locations such as startup files and the registry? How about the BIOS and the master boot sector? These areas are common infection vectors.
• Have programs such as Autoruns been used (http://technet.microsoft.com/en-us/sysinternals/bb963902.aspx)? This displays locations where programs can be set to run automatically in Windows-based systems.
• If the incident-handling team cannot stop the system from running malicious code, rebuild the system from a known good configuration to ensure the vulnerability exploited has been patched. This can be made easier if one has a sound backup policy.
As part of eradication, ensure there is no persistent malware or vulnerability that reappears after the incident-handling team removes it. Properly tuned network and host-based IDSs and IPSs can help detect new infections. If new infections occur, the incident-handling team will have to restart the process and modify the strategy.
The incident-handling team should consider using a malware analysis lab. These solutions are normally appliance- or software-based systems that take suspicious files and execute them in controlled sandboxed environments. During the execution, suspicious behavior such as modifying system files and reaching out to foreign servers across the Internet is logged. These behaviors can then be used to develop signatures for IDSs, IPSs, and malware removal tools.
For particularly challenging malware, specialists may need to be recruited. These specialists are called reverse malware engineers. Their job is to analyze malicious code and understand exactly how it works, what it targets, and how best to remove and disable it. These experts are expensive, and the organization must weigh the benefits versus the costs of simply restoring the system from a known good and hardened baseline configuration.
Phase 5: Recovery
Even if the incident-handling team appears to have eradicated all signs of malware and other exploits, organizations should consider rebuilding the system. The recovery phase brings the system back into operation. Systems should be recovered from known good backups and baseline configurations. These documents, configurations, and procedures should be part of the organization’s business continuity planning. In severe incidents, the organization may actually need to fail over to an alternate processing and storage site. If the organization is recovering from the alternate site, they will follow the disaster recovery plan. You can find more information about disaster recovery and contingency planning in Chapters 24 and 25.
Once the system is restored, the mission or business owner should verify that the incident handling was a success and the system is back to its normal state. The incident-handling team must always perform tests with assistance and approval from the system owner and system administrator. After obtaining the results and understanding any residual risk, the mission or business line owner will decide whether the system should return to operation. In organizations with a C&A process in place, the accreditation official will make this decision. The incident-handling team’s task does not end when the system is restored. Instead, the team must ensure that the organization’s continuous monitoring capability is searching for new threats and vulnerabilities constantly.
Phase 6: Review
The last phase in the incident-handling process is to prepare a follow-up report and organize a review meeting to allow the incident-handling team to share and learn from the experience. The team should present the report and recommend to management additional countermeasures to prevent recurrence.
Use the review phase to improve incident-handling procedures. The lessons learned also become feedback to improve the AT&E program for the organization. The AT&E program can be used to develop training and evaluation scenarios for all employees. A positive outcome from an incident improves morale and makes the overall information assurance posture stronger.
Some common follow-on activities from the review are a penetration test and a recertification for systems following a C&A process. The penetration test is often performed as a limited-scope assessment to ensure the same attack cannot happen again and to ensure the appropriate controls are in place to mitigate further attacks of a similar nature. For systems operating under a C&A program and accreditation, they should be re-accessed with a scope focused on vulnerability management, configuration management AT&E program, and any controls identified as deficient in the incident report. Any further deficiencies should be documented and reported to the accreditation official for mitigation or acceptance.
Further Reading
• Jones, K.J., R. Bejtlich, and C.W. Rose. Real Digital Forensics: Computer Security and Incident Response. Addison-Wesley, 2005.
• Kruse II, WG, and JG Heiser. Computer Forensics: Incident Response Essentials. Addison-Wesley, 2005.
• Nichols, R. Defending Your Digital Assets Against Hackers, Crackers, Spies, and Thieves. McGraw-Hill Education, 2000.
• Prosise, C., K. Mandia, and M. Pepe. Incident Response and Computer Forensics. McGraw-Hill Education, 2003.
• SANS Institute and Ed Skoudis. Incident Handling Guidelines. SANS, 2004.
• Schmidt, Howard A. Patrolling Cyberspace: Lessons Learned from a Lifetime in Data Security. Larstan Publishing, 2006.
• Conklin, Wm. Arthur. Introduction to Principles of Computer Security: Security+ and Beyond. McGraw-Hill Education, 2004.
• Schou, Corey D., and D.P. Shoemaker. Information Assurance for the Enterprise: A Roadmap to Information Security. McGraw-Hill Education, 2007.
• Tipton, Harold. F., and S. Hernandez, ed. Official (ISC)2 Guide to the CISSP CBK 3rd edition. ((ISC)2) Press, 2012.
Critical Thinking Exercises
1. An organization is using a cloud-based system for hosting its sensitive information in the form of customer lists and personally identifiable information. They are using Software as a Service (SaaS) customer relationship management (CRM) platform to manage their sales. Lately, their customers have been complaining about receiving calls from a competitor in another country with detailed information about their purchase histories and relationship with the organization. What is the incident response approach the organization should take?
2. An organization has adopted a bring-your-own-device (BYOD) approach for mobile devices such as smartphones and tablets. An employee’s tablet has been identified as an unauthorized bridge between the organization’s secure network and the public Internet. The organization’s incident-handling team is baffled as to how this could happen and attempts to call the employee and retrieve the device. The employee refuses to provide the device and cites their right to privacy and their ownership of the device. The owner states that they have sensitive personal information on the device and under no circumstance will they allow the organization to search the device. What can the organization do in this situation?