Another flaw in the human character is that everyone wants to build and nobody wants to do maintenance.
—Kurt Vonnegut, Hocus Pocus
Most of the time, you shouldn’t work too hard at being exceptional. You’re better off first making sure that you avoid doing anything too stupid. If you are hacked because of some unpatched hole that’s been sitting around for months, you will look stupid. Where did that hole come from? We know that no matter how secure we make our systems, new vulnerabilities will be found. Your challenge is to find and fix the holes before the attackers exploit them. The process you use is called vulnerability management . It is a process that combines both technical and administrative controls, calling upon many different aspects of security and coordinating work between different departments.
Vulnerability management is featured prominently in PCI DSS requirements , although you find it in most other audit requirements as well. Vulnerability management can be a real grind, but it’s an important and powerful process. Enough to warrant a whole chapter devoted to it.
The activities associated with vulnerability management can be broken down into four general categories: hardening, scanning, prioritizing, and patching, as shown in Figure 14-1.
Figure 14-1. Vulnerability management activities
In some audit and control regimes, these are wholly defined separate controls, but for this chapter, we are combining them into a single practice.
Organizing Vulnerability Management
Let’s begin with the policy to define what we want vulnerability management to be within the organization.
Sample Vulnerability Management Policy
To reduceexploitable security vulnerabilitiesin systems for ORGANIZATION in a timely manner, there will be:
Hardening standards for all asset classes as a requirement before they go live
Monitoring of vendor vulnerability notifications for critical applications
Monthly internal vulnerability scanning of local area networks of critical systems
Quarterly external vulnerability scanning of entire organization’s Internet perimeter
Annual penetration testing of entire organization’s Internet perimeter
Application security testing for critical custom applications for every major release
Vulnerability scoring and prioritization of all discovered vulnerabilities
Patching and remediation of all high and medium scored vulnerabilities on critical assets within 30 days of discovery
Responsibility for these activities will be assigned and tracked by the ISMS committee. Quarterly vulnerability reports and vulnerability management schedules will be reviewed by the ISMS committee to monitor the ongoing state of risk within the organization.
Systems covered by this policy must be tested to meet these standards before being released onto operation. This includesphysical and virtual machines, both locally and offsite in public hosting or cloud providers.
This policy will be reviewed on an annual basis and may change regulations or requirements.
Vulnerability Management Breakdown of Responsibilities
Assigning all of these work activities sounds like a good use for a RACI matrix. Table 14-1 is an example of how you might want to break this down.
Table 14-1. Vulnerability Management Breakdown of Responsibilities
Activity | Security | IT | Asset Owner | ISMS committee |
|---|---|---|---|---|
Hardening standards | A | R | C | I |
Vulnerability notification | A | R | C | I |
Internal scanning | A | R | C | I |
External scanning | R,A | C | C | I |
Penetration testing | R, A | C | C | I |
Application security testing | R, A | C | C | I |
Scoring and prioritization | R, A | C | C | I |
Patching and remediation | A | R | C | I |
Hardening Standards
Before you begin managing vulnerabilities, you need to know what is considered secure for your organization. Like everything else, it is informed by risk and business needs and described with administrative controls. These standards are the you must be this tall to ride checkpoint for any device or service going live in your organization. Next is an example of a base standard for hardening. You’ll see that it calls out for more standards beneath it to describe the specifics. These standards form the bedrock for scanning baselines, risk analyses, implementation work, and technology acquisition decisions. This is a good example of the clarity and utility that strong administrative controls can bring.
Sample Hardening and Vulnerability Management Standard
The IT department and the Security department will maintain this approved standard for performing vulnerability management. This standard will address identifying, remediating, and documenting technical vulnerabilities for the following types of assets:
Security devices
Internet-facing servers and network devices
Corporate servers and network devices
Accounting systems and network devices
Desktop computers and laptops
Virtual or public-cloud servers and services
The IT department will be responsible for maintaining a configuration management and device-hardening standard for ORGANIZATION servers. This hardening standard will include descriptions of
Baseline firmware configuration
Baseline operating system configuration
Baseline software packages installed
Baseline software packages configuration
Hypervisor configuration
Public cloud access, host, and networkconfigurations
Approved resource monitoring and management tools
Administrative access requirements
Approved network services
Documentation requirements
These standards will be updated no less frequently than annually. The IT department and the security department will also update these standards based on new technology, organizational changes, and risk changes.
This policy will be reviewed on an annual basis and may change regulations or requirements.
How to Fill in the Hardening Standards?
Some IT departments look to the security team to lead in defining the standards. While it is important that they have buy-in to what is in the standard, it is reasonable to give them a starting place. Many software, hardware, and Internet service manufacturers include secure configuration guides with their products. In general, you should be hesitant in working with technology vendors who do not offer such guidelines. Some of your compliance requirements may specify specific standards to meet for scoped devices. In addition, there are several independent sources of hardening standards. Here is a short list of some to consider:
Center for Internet Security (CIS) Host Benchmarks
National Security Agency (NSA) Secure Configuration Guides
https://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml
National Checklist Program Repository
Department of Defense Information Assurance Support Environment Security Requirement Guides
Microsoft Server Hardening
Apple Security Guide
Amazon Web Services Security Best Practices
https://aws.amazon.com/whitepapers/aws-security-best-practices/
Microsoft Azure Security Practices
https://azure.microsoft.com/en-us/documentation/articles/best-practices-network-security/
You will find literally hundreds of pages of ideas here. Over time, you will probably add items to your lists as new risks and compliance issues are uncovered. Stepping away from the specifics, here are some general things you should consider including in your hardening standard:
Remove or rename default pre-installed accounts
Change default passwords and shared secrets
Protect and log administrative interfaces
Prevent unknown code from running
Disable unnecessary services
Disable poorly encrypted services
Disable poorly authenticated services
Lastly, if you have systems that don’t require Internet connections, then don’t connect them to the Internet. It seems the default expectation for every device and application is that it be granted full Internet connectivity. If it isn’t necessary for the service the device is providing, disable Internet access. You should question and reconsider any vendor that argues with you on this point, especially when it comes to mandatory Internet-connected maintenance links back to the vendor’s network. Some devices may not even allow you to disable their Internet connectivity. For these systems, you can look into either giving them a fake or non-existent default gateway. That’ll usually prevent them from connecting to the Internet. If systems are disconnected from the Internet, there should still be mechanisms in place to have them patched in a timely manner. This can be as simple as temporarily connecting them for patching or using portable media to deliver updates.
Vulnerability Discovery
Now that you have standards, how do you know they’re being met? The prudent answer is you should assume they aren’t and check for yourself regularly. This means a schedule and assigned responsibility to do the checking. The standards and procedures for this can include the following:
Vulnerability discovery frequency scans
Type of discovery scan
Who is authorized and is responsible to scan
Who receives the results
How results are protected (as they contain confidential information about your organization’s weaknesses)
As you can see, vulnerability management can be a grind. It is necessary to do it this way to be thorough and prevent a potential surprise attack against systems you assumed were impregnable.
Vulnerability Notification
There are many technical tools that you can use for vulnerability discovery, but they’re not required. You could do vulnerability discovery manually with an accurate, detailed inventory and an updated list of published vulnerabilities. However, with the average of several of machines to every employee, this gets hard to manage very quickly. It still is a prudent to subscribe to e-mail notification or Really Simple Syndication (RSS) lists of vulnerabilities from your major vendors. Responsibility for reading these notifications and responding to them needs to be assigned as well. For example, a security analyst could be on a “Microsoft Patch Tuesday” e-mail notification list and be responsible for alerting IT about upcoming urgent problems. In addition to software manufacturers, there are many independent sources of vulnerability notifications. If you are member of an ISAC or a security organization, you can get alerts on industry vulnerabilities there. Here are some other good resources:
National Vulnerability Database
US-CERT
Discovery Scanning
As discussed back in the chapter that introduced assume breach, one of the key problems in information security is knowing what you have and where it is. How can you have a complete picture of your vulnerabilities are if you can’t identify all of your assets? In addition, how do you know if your published standards are being followed? Therefore, the primary step in vulnerability management is doing discovery scanning. This involves using technical utilities to sweep your systems to identify computers, appliances, services, software, and data.
One of the most basic of these tools is network port scanner software , which runs on one or more hosts and sweeps the network looking for services running. Network scanners can also do fingerprint-based pattern matching to identify known software packages. This is done with a series of probes to a list of addresses and automated analysis of the responses.
Scanning can be done internally, on your private local area networks, and externally on the Internet perimeter. External network scanning needs to be done from an Internet host outside of your organization’s perimeter. External scanning can give you a “hacker’s eye view” of how your Internet-services appear to attackers. You may be surprised as to what is visible and what is not visible from this perspective, which makes this kind of scanning always worth doing.
One of the most popular port scanners is Nmap , an open-source network scanner that’s been around for almost two decades. Nmap runs on wide variety of operating systems and is available at https://nmap.org/ . This is how you can use Nmap to get an inventory and footprint of a network:
$ sudo nmap 192.168.0.1-16Starting Nmap 6.01 ( http://nmap.org ) at 2016-03-13 14:07 PDTNmap scan report for 192.168.0.1Host is up (0.0063s latency).PORT STATE SERVICE80/tcp open http1900/tcp open upnpNmap scan report for 192.168.0.10Host is up (0.0025s latency).PORT STATE SERVICE22/tcp open sshNmap scan report for 192.168.0.14Host is up (0.00011s latency).PORT STATE SERVICE3689/tcp open rendezvous
There are also many other free and commercial tools out there that can do automated asset and configuration discovery. There are some tools directly available from vendors that can do configuration scanning and analysis of their products and services. For example, many network device manufacturers have device configuration scanners and many large cloud-hosting services provide inventory tools.
The key is to do regular discovery scanning so you have an as accurate and freshest picture as you can. This entails having a recorded inventory and standard to compare the scans against. Changes in your vulnerability and inventory are worth tracking as well. If a system has a history of vulnerabilities, perhaps a new design is needed instead of frequent patching. Scanning frequency is important as well. I have seen some organizations do daily scanning and analysis of their rapidly changing environment. You have to be careful, though, as some network scanners can create a significant load on the network. Having an accurate scope of the scanning can help reduce noise and time in a scan as well. You should only scan the things that you need to scan, instead of trying to saturate the network with probes. The following are things to look out for when discovery scanning:
Unmanaged hosts—such as equipment that’s fallen off IT management’s radar—that could be missing patches and critical controls, such as antivirus.
Out-of-office laptops and mobile devices that occasionally return from the field full of vulnerabilities and malware.
Non-standard equipment—such as Wi-Fi hotspots—that people have sneaked in from home and plugged into the corporate network.
Virtualized guests that can suddenly appear on your network or within your cloud environment. With virtualization, it’s easy to spin up boxes and forget about them.
New devices that you didn’t know were network aware until someone plugged them in, like the proverbial Internet refrigerator.
Spying devices planted on your network by the bad people. You definitely want to try to find these.
Vulnerability Scanning
The next step after discovery scanning is to do vulnerability scanning. Many discovery scanners have vulnerability scanning functions as well. Like network port scanners, vulnerability scanners can actively send network probes at a server and attempt to decipher the responses. The scanner functions in two ways. One way is to determine the exact version of software being run and then match that software to published vulnerability lists. The second way is to do a harmless attempt at a hacking attack and see if it succeeded. This method is slightly dangerous but far more effective in determining if there is a potential problem. At http://sectools.org/tag/vuln-scanners/ , you can see a large, but not necessarily complete, list of vulnerability scanners. There are also vulnerability scanners that specialize in just scanning common types of services, such as database scanners or web scanners. This is how the free, open source, web scanner Nikto ( https://cirt.net/Nikto2 ) looks in action:
$nikto-2.1.5>perl nikto.pl -host 192.168.0.5- Nikto v2.1.5---------------------------------------------------------------------------+ Target IP: 192.168.0.5+ Target Hostname: scandog.chaloobi+ Target Port: 80+ Start Time: 2016-08-15 15:40:17 (GMT-7)---------------------------------------------------------------------------+ Server: Apache/2.2.29 (Unix)+ Server leaks inodes via ETags, header found with file /, inode: 78545071, size: 633, mtime: 0x526692b64c840+ The anti-clickjacking X-Frame-Options header is not present.+ Allowed HTTP Methods: POST, OPTIONS, GET, HEAD, TRACE+ OSVDB-877: HTTP TRACE method is active, suggesting the host is vulnerable to XST+ 6545 items checked: 992 error(s) and 4 item(s) reported on remote host+ End Time: 2016-08-15 15:41:26 (GMT-7) (69 seconds)---------------------------------------------------------------------------+ 1 host(s) tested
Some vulnerability scanners can work passively, by examining captured network traffic, otherwise known as sniffed traffic. The scanner looks at the network requests and responses and tries determining what software versions are in use. Another passive type of vulnerability scanner is an agent-based one, which is a small piece of software running on every host on the network. This type of scanner can be very accurate since it has direct access to the local software and configuration, at the expense of having something running on every box in your organization. Some passive agents also watch the local network traffic to help identify unagented machines that haven’t been inventoried yet.
Vulnerability scanning of your network is a requirement of many audit regimes. PCI DSS requires at least quarterly vulnerability scans both internally and externally. In general, scanning frequency should be based on how frequently your environment changes and how often new vulnerabilities in popular software are discovered. Quarterly is just an industry average. Some organizations do monthly, weekly, or even daily scanning.
Some organizations choose to have outside security vendors do their vulnerability scanning, especially the external scanning. This is useful because of the expense and expertise required to do a thorough and accurate vulnerability scan. Some of the best commercial vulnerability scanners are not cheap and need a trained operator to interpret the results. The PCI DSS and many regulatory regimes require that a qualified vulnerability assessor perform the external vulnerability scans . This is not just because of the expertise needed but it also provides an independent third-party opinion of the vulnerabilities found.
Penetration Testing
Going beyond vulnerability scanning is penetration testing. Like the vulnerability scanner that can try a harmless version of a hacking attack to test for a vulnerability, penetration testing takes this a step further. A penetration test is an active attempt to break into your network. Some people conflate vulnerability scanning and penetration testing, but they are not the same thing. For one thing, penetration tests are usually an order of magnitude more expensive than a vulnerability scan. Penetration tests can also include techniques not normally found in a vulnerability test, including password-guessing attacks, social engineering phishing tests, wireless scanning, and even physical break-in attempts. It’s one thing to point a software-scanning tool at your web server and another to have a security tester try to burgle your premises.
Good penetration testers combine techniques, such as rooting through your trash (dumpster diving) to determine potential targets, applying physical penetration to plant network bugging devices, and using custom-built hacking tools. A fun example of how this works is the short-lived TV series Tiger Team, which is available online just a search engine query away.
An annual penetration test by a qualified third-party firm is a PCI DSS requirement. These are different from quarterly penetration test, although often the same assessor can perform both. Regarding qualification, the PCI DSS requires organizations to use vendors certified in PCI DSS scanning. These vendors are called approved scanning vendors(ASV) and are listed on the PCI DSS site at https://www.pcisecuritystandards.org/assessors_and_solutions/approved_scanning_vendors .
Dynamic Application Testing
Another form of security testing more specific to in-house developed software is application security testing . There are many types of application security testing. Some firms specialize in testing specific types of platforms, such as web applications or mobile apps. Another way to do vulnerability testing is to offer bug bounties and to crowdsource the testing of your software. Bug bounties involve offering prizes (usually cash) to whoever submits a valid vulnerability (and keeps it quiet). Some businesses specialize in organizing and running your bug bounty program for you.
If you do sell or provide software services for an in-house developed application, you should define and publish a process describing how you will receive and deal with vulnerabilities discovered outside of your organization. This is a major topic that is more in-line to secure application development, but it is worth mentioning that it is something you should investigate. A good starting point on building this process is at https://blog.bugcrowd.com/security-vulnerability-submission/ .
Prioritization and Risk Scoring
Now that you’ve been doing all this vulnerability scanning, you need to have a process in place to deal with this influx of data. If you’ve never done vulnerability scanning before, you’ll quickly discover the tsunamis of data a single scan produces. Furthermore, most scanning tools and vendors will include a priority score along with their results. Usually these are based on the Common Vulnerability Scoring Standard (CVSS ) that rates vulnerabilities from the lowest 1 to the highest 10. The CVSS uses factors like exploitability, complexity, and necessary authentication to service the scope. Its calculation method is explained more at https://www.first.org/cvss .
Problems arise having a third party use CVSS because often they don’t understand your environment and the potential interactions vulnerabilities can cause. I have seen harmless vulnerabilities scored as 10, and have had penetration testers break in using two vulnerabilities scored as a 3. Furthermore, research has shown that CVSS is not a good reflection of real world likelihood and impact of attack.
The conclusion to draw from this is that regardless of what the scanner or vendor reports, you need to do your own analysis and prioritization of the vulnerabilities discovered. Obviously, anything not meeting your published standards needs to be addressed. For PCI DSS covered entities, you are also required to patch any vulnerability scored higher than a CVSS level 4. Even then, you may still have a long list of items to fix. What should come first?
Higher Priority
The first things you should look at are the vulnerabilities that are easily exploitable over the Internet. If there is an existing hacking tool that a script kiddie can just push a button and execute from a Wi-Fi café that takes down your site, then you need to fix it immediately. Many scanning tools and third-party assessors will provide this type of exploitability information in their vulnerability report. If they don’t, reconsider your scanning vendor. You can also check some known lists, like the Exploit database at https://www.exploit-db.com/ . You’ll find that these same vulnerabilities also fall into the category of things you need to fix so you don’t look stupid and/or negligent. If the whole Internet is in a panic about the new Ahab Hole in BigWhale Servers, then it’s prudent to get on that quickly.
On the inside network, the highest priority items are going to be the common services that touch the outside world in some way. Right now, this translates into Internet browsers and their associated scripting languages. A medium-rated vulnerability that is running on 95% of your user’s browsers that surf the Internet all day long is a bigger priority than a highly rated vulnerability on a seldom-used service that never talks outside the local network.
Lastly, you should use organizational factors like the impact calculations you’ve done during your risk analysis to prioritize vulnerability remediation work. This is where what you’ve determined may diverge from what an outside-scored vulnerability tells you. You know which things need to be available 24/7 and may hold confidential data, as well as which things you don’t care as much if they are breached. An outside party’s scoring vulnerabilities may not. The CVSS calculation actually has variables for this including collateral damage, prevalence of vulnerability and the impacts. You can use these to refine your scores. The calculator at https://www.first.org/cvss/calculator/3.0 is helpful for doing this.
Lower Priority
Things that are deep in your network, protected by multiple layers of controls with little or no connection to the outside world can probably be prioritized lower. For example, consider a web server, an application server, and a database server, each separated by a proxy firewall. The web server touches the Internet, so it would have high priority. You could deprioritize a firewalled database with no direct outside access.
You should also be aware that low priority vulnerabilities often do not remain low priority forever. The cycle of innovation in the hacking world moves quickly. A vulnerability could be scored low because it requires a complicated manual attack and the service only runs on local area networks. The following week, a new worm could be released that includes an automated exploit of that vulnerability and the worm is designed to run on inside networks.
More Food for Thought
Beyond re-scoring and prioritizing vulnerabilities, the security team should also be doing analysis on the nature of the problems discovered. What caused these vulnerabilities to appear? Is there a breakdown in automatic patching routines ? Are systems not being built according hardening standards? Should new controls or processes be put in place?
Patching
With the deluge of vulnerabilities and the rapid proliferation of attack tools, patching should not be a painful, one-off exercise for any organization. Regular patching of every system should be a procedure and performed regularly. This includes the capability to patch the fragile, can’t-ever-go-down systems that people rely on. It’s a safe bet that a new high-priority vulnerability will be found and the IT department will have to scramble to patch. Work out and test the plan now before you have to do it under fire.
A deadline for patching based on priority should be documented and communicated. A common standard is thirty days maximum for high priority vulnerabilities, although you might want to be able to go faster. When something scary is running around, you may have a window of days if not hours to get things locked down.
Having spent a large portion of my career in and around IT operations, I appreciate just how much work is involved in patching. It definitely is the hardest part of the grind of vulnerability management. This is where having clear prioritized vulnerabilities and a well-functioning change control system in place can help your organization patch fast.
Some vulnerabilities can be a real beast to fix and you may find that they can’t be fixed within your defined schedule. This is where you need to document your best efforts and work them like small projects. You never want to throw your hands up and just forget patching because we’re replacing that system next year anyway. This is how you end up looking like a monkey when your organization is hacked. If nothing else, write up a plan, have the ISMS committee buy off on the risk acceptance, and set deadlines to have the vulnerability remediated.
Scan Again
An often-overlooked aspect of patching is patch verification. An essential part of the remediation process is to rerun the vulnerability tests to ensure that the patch actually closed the hole. Only then can a vulnerability be considered closed.