Appendix D. Security Incident Handling
Chapter 2 outlined many threats against IP (and L2 Ethernet) networks. Chapters 4 through 7 described a wide variety of techniques available to mitigate these threats. Although this book focuses on IP network traffic plane security, many other threats exist that aim to exploit vulnerabilities in host operating systems and application software. Hence, network operational security must consider both network-based attacks and host-based attacks.
This appendix focuses on security incident handling; that is, the method by which you prepare for and respond to active host-based or network-based attacks. The industry best common practice (BCP) for incident response handling includes a six-phase approach, which is described here. In addition, this appendix provides a brief summary of Cisco product security and several industry incident response teams and network operators’ groups.
Security operators are also recommended to consider building their own security operations center (SOC). This appendix does not cover SOC designs or operations. More information on this topic can be found in the Cisco white paper “How to Build a Cisco Security Operations Center,” available on Cisco.com. For more information on security incident handling, see the “Further Reading” list at the end of the appendix.
Six Phases of Incident Response
Malware, including viruses, worms, and distributed DoS attacks, may adversely impact legitimate traffic flows and network infrastructure, including the wider Internet, within minutes or even seconds. Consequently, the speed with which you recognize and respond to attacks is critical to minimizing the impact of an attack. When an effective incident response plan is not available, networks are at increased risk.
To reduce incident response times, you must proactively establish incident response procedures within an operational security framework, as opposed to simply reacting to events. This also requires monitoring for security events so that attacks can be quickly detected. The industry BCP for incident response handling includes a six-phase approach, which is illustrated in Figure D-1. In adopting these phases (or steps) within your security operations framework, you may significantly reduce response times and improve the mitigation effectiveness against attacks. In addition, this six-phase approach has proven capable of serving well for addressing both existing and emerging threats.
Figure D-1. Six Phases of Incident Response
Let’s review each of these six phases.
Preparation
Being prepared significantly improves your ability to respond to attacks and, therefore, improves response time, improves mitigation effectiveness, and reduces network downtime. Without preparation, you are often left to employ highly reactive and perhaps misguided actions in the event of an attack. Preparation is considered the most difficult and costly phase of the six-phase approach, yet it represents the most important of the six phases because it lays the foundation for the remaining phases. The preparation phase involves a number of distinct components, which are described next.
Understand the Threats
As outlined in Chapter 2, there are a variety of methods by which attackers may target IP networks and devices. Further, threats may differ due to the variety of IP networks deployed, including product mix, network topology, traffic behavior, and organizational mission (for example, SP versus enterprise). Understanding the threats against your specific network will help you to assess your risk, mitigate the risk to acceptable levels, and classify attacks once detected.
Deploy Defense in Depth and Breadth Security Strategies
IP routers and network devices today support a wide variety of security mechanisms to detect, prevent, and mitigate attacks, as outlined in Chapters 4 through 7. These mechanisms must be proactively deployed, however, because implementing them in the midst of an attack may place the network at even greater risk given the potential for unintended consequences such as misconfiguration errors and collateral damage. For example, implementing certain features may cause router performance degradation. When this is not well understood, implementing a feature in the midst of an attack without prior understanding of feature impact can cause more problems than the attack itself.
Performance impacts, if any, depend on different factors, as outlined in Chapters 1 and 3. Therefore, to harden the network infrastructure and minimize the risk of an attack (as well as harmful side effects resulting from reactive configuration changes), defense in depth and breadth strategies should be proactively deployed. An example of where this is critical is preprovisioning, testing, and establishing a usage procedure for the mechanisms required to implement remotely triggered black hole (RTBH) filtering (such as deploying a static route to Null0 on all edge routers and deploying a BGP trigger router as described in Chapter 4). Deploying up-to-date software versions that include fixes for disclosed security vulnerabilities is another proactive step you should take to mitigate the risk of known vulnerabilities.
When emergency software upgrades are required, understanding available flash and dynamic memory as well as having prepared procedures for performing upgrades reduces the risk of errors and collateral damage. Understandably, deploying infrastructure security can be difficult because it affects many network devices, each of which potentially has its own limitations and platform-specific dependencies. Further, there is a cost associated with deploying security measures, which may include administrative overhead, operational inconvenience, and router scale and performance impacts. The cost of applying security measures needs to be weighed against the potential risks. Organizations (not just security operators) must understand the risks and the cost of applying security measures to mitigate the risk to acceptable levels.
Establish Well-Defined Incident Response Procedures
As previously described, you should prepare the network in advance with any preconfigurations necessary for attack mitigation, as opposed to configuring in real time during an incident. Once you have done so, it is then imperative that you establish a playbook that defines the roles and responsibilities of everyone on the incident response team. Well-defined procedures must be established and training drills must be conducted. This not only helps people understand their roles, but brings to light any areas of question, allowing for procedural modifications where required. Further, these incident response procedures must consider the associated performance impact, if any, of enabling a security feature for all applicable network equipment before deploying it. Without knowing the performance impacts, if any, applying a mitigation technique such as an ACL, for example, may actually have a more adverse impact on the network than the attack itself. The established incident response procedures must take these factors into consideration as previously stated.
Establish an Incident Response Team
Because security attacks threaten network availability, the incident response team should include both network and security operators. They must be well trained and versed in their roles during times of attack. Once attacks occur, it is too late to begin identifying who is doing what, where, and when. The incident response team owns the six phases of incident response and is responsible for executing against each of them. Further, the incident response team should also maintain contact information for all external network peers. Many attacks are sourced from external networks. Hence, it is important to maintain emergency contact information and understand how they may be able to assist in attack mitigation. For SPs, an Inter-NOC (INOC) Dial-By-ASN (DBA) Hotline is also available to facilitate real-time communication among the SP community. For more information, refer to http://www.pch.net/inoc-dba/ and http://www.pch.net/technology/operations.php#3.
Identification
In order to mitigate an attack, it must first be detected and identified. Detection requires visibility into network activity, threats, and traffic patterns. Without such network visibility, you are left with an incomplete view of network traffic and events. This significantly increases the time to repair (or mitigate) depending upon the root cause diagnosis. As stated in the previous section, detection time is critical to containing the impact of an attack. IP routers support a wide variety of tools that provide network visibility and anomaly detection, as outlined in Chapter 6. These include but are not limited to SNMP polling and traps, syslog messaging, NetFlow telemetry, and various other router health statistics such as those related to CPU and memory utilization and feature performance. Such network telemetry is considered a network security best practice and should be defined and deployed as part of the preparation phase previously outlined.
Further, to detect network anomalies and potential security events, you must first understand the baseline network activity and traffic patterns during normal network operating conditions. The comparison of real-time network conditions against the established baseline is the very nature of the identification phase. For more information on network telemetry and event identification, refer to the Cisco Networkers 2005 session SEC-2102 entitled “Detection and Classification of Network Traffic.”
Classification
Classification provides the context for further action (in other words, the traceback and reaction phases, discussed next) once a network fault or anomaly is identified. Network events may be caused by any number of sources, as outlined in Chapter 2, including both intentional and unintentional threats. Classification is about diagnosing the problem cause, severity, and scope of the threat. For example, does the threat affect a single device or the wider network infrastructure, and what damage is it causing? Classification also relies on network telemetry to gain network visibility. Whereas the previous identification phase collects and establishes trends for network activity and traffic patterns, the classification phase correlates the observed network activity and events in order to isolate problem cause and determine a root cause.
Traceback
After an attack has been detected and classified, you need to identify and locate the source(s) of the attack. Attacks that use spoofed IP source addresses are the most difficult to trace back to their origin. Source traceback for spoofed attacks can be arduous. Deploying antispoofing protection as part of the preparation phase minimizes such threats and can significantly reduce the associated operational expenses of dealing with such attacks.
IP routers support a wide variety of tools that facilitate source identification and traceback of an attack, including but not limited to classification ACLs, NetFlow, IP Source Tracker, and the ICMP backscatter traceback technique. If an attack originates externally, then it must be traced back to the point(s) of ingress at the network edge. Once it has been traced back to your network edge, the pre-established contacts with your peer networks (as discussed earlier in the section “Establish an Incident Response Team”) become useful for gaining mitigation support from external peer networks. Traceback must also consider whether multiple paths exist to the external peer from which the attack originates.
Reaction
Once an attack has been identified, classified, and traced back to the source(s), you may need to explicitly mitigate it. If the attack is insignificant or inconsequential, you may decide not to do anything. Chapters 4 through 7 describe a variety of mechanisms to protect and mitigate attacks against IP networks and IP routers. No single technique can be identified as the best approach to mitigate all of the many different threats. The effectiveness of each technique is dependent on specific network environments such as product mix, network topology, traffic behavior, and organizational mission. Nevertheless, you should avoid deploying techniques that have not been previously defined within the established incident response procedures documented during the preparation phase previously described. Without understanding the potential impacts, if any, applying a mitigation technique may make the problem worse. Further, attacks should be mitigated as close to the source or ingress point(s) as possible. Otherwise, a mitigated attack may still have the potential to cause collateral damage on intermediate network devices.
Post-Mortem Analysis
After the dust settles, your incident response team should analyze the root causes of the incident, and fine-tune existing incident response procedures. This should result in changes to incident response procedures, where required, and potential changes in the baseline network security configuration. In this way, the incident response team and procedures are continuously evolving and providing greater resistance to known and emerging threats.
Cisco Product Security
The Cisco Product Security Incident Response Team (PSIRT) is a dedicated, global team that manages the receipt, investigation, and public reporting of security vulnerability-related information, related to Cisco products and networks. PSIRT works with Cisco customers, independent security researchers, consultants, industry organizations, and other vendors to identify possible security issues with Cisco products and networks. Responses can range from Release Note Enclosures (RNE), which are visible to customers via BugToolkit on Cisco.com, to Security Advisories, depending upon a number of factors.
Anyone who has a product security issue is strongly encouraged to contact PSIRT directly. To report security-related bugs in Cisco products, or to get assistance with security incidents involving Cisco products, send an e-mail to [email protected] for nonemergency issues or [email protected] for urgent matters. Cisco PSIRT may also be contacted via the PSIRT Security Hotline by dialing 877 228-7302 or 408 525-6532. Alternatively, if you are under active security attack or have more general security concerns about your Cisco network, you can contact the Cisco Technical Assistance Center at 408 526-7209, 800 553-2447, or by locating country-specific contact information. Cisco worldwide contact information is available at http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml. The technical support agents will escalate to the proper PSIRT personnel to assist you. For more information, refer to the following section, “Cisco Security Vulnerability Policy.”
Cisco Security Advisories are available via the following methods:
• Cisco’s Internet web portal at http://www.cisco.com/en/US/products/products_security_advisories_listing.html.
• E-mail via [email protected]. Anyone interested may subscribe to this e-mail list using the procedures described in the “Subscribing to the Customer Security Announce Mailing List” section of the Cisco Security Vulnerability Policy, described in the following section.
• PSIRT RSS feeds available via Cisco.com. These feeds are free and do not require any active Cisco.com registration. Information for subscribing to RSS feeds is found at http://www.cisco.com/en/US/products/products_psirt_rss_feed.html.
Major Cisco Security Announcements are also available at http://www.cisco.com/security/announcements.html.
Cisco Security Vulnerability Policy
Cisco’s policy for receiving and responding to products and services security vulnerabilities is posted at http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html.
Cisco Computer and Network Security
If you want to report a computer or network security-related incident involving the Cisco corporate network, please contact the Cisco Computer Security Incident Response Team (CSIRT) by sending an e-mail to [email protected].
Cisco Safety and Security
To report an issue or inquire about Cisco’s safety and physical security program, including the protection of company employees, property, and information, please call 408 525-1111 or send an e-mail to [email protected].
Cisco IPS Signature Pack Updates and Archives
Cisco IPS Active Update Bulletins are posted at http://www.cisco.com/security.
Cisco Security Center
Visit the Cisco Security Center site for information on emerging threats and the Cisco network IPS signatures available to protect your network. The Cisco Security Center is available at http://www.cisco.com/security/center/home.x.
You can also find Cisco Applied Intelligence Response documents at the Cisco Security Center site. Cisco Applied Intelligence Responses (AIRs) provide identification and mitigation techniques that can be deployed on Cisco network devices. As applicable, Cisco IOS access control lists, Cisco Intrusion Prevention System (IPS) signatures, Control Plane Policing, and firewall rules are among the techniques discussed in the AIR.
Cisco IntelliShield Alert Manager Service
Cisco Security IntelliShield Alert Manager Service provides a comprehensive, cost-effective solution for delivering the intelligence that organizations need to identify, prevent, and quickly mitigate IT attacks. IntelliShield Alert Manager Service is a customizable, web-based threat and vulnerability alert service that allows security staff to easily access timely, accurate, and credible information about vulnerabilities that may affect their environments, without conducting time-consuming research. Registration is required. For more information, refer to http://www.cisco.com/en/US/products/ps6834/serv_group_home.html.
Cisco Software Center
The latest Cisco software is posted to the Cisco Software Center at http://www.cisco.com/kobayashi/sw-center/. Access requires a Cisco.com username and password.
Industry Security Organizations
There are a number of leading industry and government security organizations that help the industry and Internet community deal effectively with emerging security threats. Contact information for Computer Security Incident Response Teams (CSIRT) that have responsibility for an economy or country is available at http://www.cert.org/csirts/national/contact.html. An interactive map is also available at http://www.cert.org/csirts/national/ to locate CSIRTs around the world with national responsibility.
Industry forums include but are not limited to the following:
• CERT/CC (Computer Emergency Readiness Team/Coordination Center)
http://www.cert.org/
• Cisco PSIRT (Product Security Incident Response Team)
http://www.cisco.com/go/psirt
• FIRST (Forum of Incident Response and Security Teams)
http://www.first.org/
• IETF OPSEC (Operational Security Capabilities for IP Network Infrastructure)
http://www.ietf.org/html.charters/opsec-charter.html
• SANS Internet Storm Center
http://isc.sans.org/
• IT-ISAC (Information Technology – Information Sharing and Analysis Center)
https://www.it-isac.org/
• ITU Cybersecurity Gateway
http://www.itu.int/cybersecurity/itu_activities.html
• NSP-SEC Forum
http://puck.nether.net/mailman/listinfo/nsp-security
• SANS (SysAdmin, Audit, Network, Security) Institute
http://www.sans.org/
• TERENA (Trans-European Research and Education Networking Association) TF-CERT
http://www.terena.nl/tech/task-forces/tf-csirt/
• US-CERT
http://www.uscert.gov/
• World Wide ISAC (Information Sharing and Analysis Center)
http://www.wwisac.com
Regional Network Operators Groups
In addition to the industry security associations, a number of leading industry operator forums help the industry and regional Internet communities to effectively deal with network operational issues, including operational security (OPSEC). Many have regular meeting forums, Internet portals, and e-mail mailing lists that offer open participation to all interested parties.
• AFNOG (African Network Operators’ Group)
http://www.afnog.org/
• APRICOT (Asia Pacific Regional Internet Conference on Operational Technologies)
http://www.apricot.net/
• APOPS (Asia Pacific Operators Forum)
http://www.apops.net/
• CNNOG (China Network Operators’ Group)
http://www.cnnog.org/index-e.html
• FRnOG (French Network Operators Group)
http://www.frnog.org/
• DENOG (German Network Operators Group)
http://www.denog.de/
• IE-NOG (Irish Network Operators Group)
http://www.ienog.org/
• JANOG (Japan Network Operators’ Group)
http://www.janog.gr.jp/index-e.html
• MENOG (Middle East Network Operators Group)
http://www.ripe.net/meetings/menog/
• NANOG (North American Network Operators’ Group)
http://www.nanog.org/
• NAPLA (Latin America NAP Regional Interconnection Forum)
http://lacnic.net/en/eventos/lacnicx/napla.html
• NZNOG (New Zealand Network Operators Group)
http://www.nznog.org/
• PacNOG (Pacific Network Operators Group)
http://www.pacnog.org/
• RIPE NCC (Réseaux IP Européens Network Coordination Centre)
http://www.ripe.net
Note
RIPE also hosts the European Operators Forum (EOF) as part of the RIPE meetings.
• SANOG (South Asian Network Operators Group)
http://www.sanog.org/
• SwiNOG (Swiss Network Operators Group)
http://www.swinog.ch/
• UKNOF (UK Network Operators’ Forum)
http://www.uknof.org.uk/
Further Reading
Dobbins, R. “Detection and Classification of Network Traffic.” Session SEC-2102. Cisco Networkers 2005. June 2005.
Gemberling, B., C. Morrow, and B. Greene. “ISP Security – Real World Techniques: Remote Triggered Black Hole Filtering and Backscatter Traceback.” NANOG. http://www.nanog.org/mtg-0110/greene.html.
Greene, B., and R. Dobbins. “ISP Security 101 Primer.” NANOG. http://www.nanog.org/mtg-0602/greene.html.
Kaeo, M. “Current Operational Security Practices in Internet Service Provider Environments.” RFC 4778. IETF, Jan. 2007. http://www.ietf.org/rfc/rfc4778.txt
Morrow, C., and B. Gemberling. “How to Track a DoS Attack.” NANOG. http://www.secsup.org/Tracking/.
Parmakovic, D. “Service Provider Security.” Cisco white paper. http://www.cisco.com/web/about/security/intelligence/sp_infrastruct_scty.html.
Stewart, J. “Vulnerability Disclosure.” Cisco Executive Thought Leadership Research Perspective. http://tools.cisco.com/dlls/tln/page/media/perspectives/detail/ep/2006/john-stewart-01.
“How to Build a Security Operations Center.” Cisco white paper. http://www.cisco.com/en/US/netsol/ns341/ns121/ns310/networking_solutions_white_paper0900aecd80598c16.shtml.
“ISACs.” Cisco Incident Response Support. http://www.cisco.com/web/about/security/security_services/ciag/incident_response_support/ISACs.html.
“NANOG Security Curriculum.” NANOG. http://www.nanog.org/ispsecurity.html.
“New Rapid Response Strategy Helps Security Services Firm Block Emerging Network.” Cisco Case Study. http://www.cisco.com/en/US/products/ps6542/products_case_study0900aecd803fc82a.shtml.