Contrary to popular belief, logging and monitoring are not constrained to Requirement 10 but, in fact, pervade all 12 of the PCI DSS requirements; the key areas where logging and monitoring are mandated in PCI DSS are Requirement 10 and sections of Requirement 11.

As we mentioned previously, for those who are used to thinking of security as prevention or blocking, the benefits of monitoring might need to be spelled out explicitly, Before we go into describing what security monitoring measures are prescribed by PCI, we will do exactly that.

So, why monitor?

First comes situational awareness. It simply means knowing what is going on in your network and on your systems and applications. Examples here are Who is doing what on that server?, Who is accessing my network?, or What is that application doing with card data? In addition, system logging helps you know not only what is going on but also what was going on; a vital component needed for investigations and later incident response.

Next comes new threat discovery, which is simply knowing the bad stuff that is happening. This presents one of the major reasons to collect and review logs as well as to deploy intrusion detection systems (IDSs).

Third, logging helps you to get more value out of the network and security infrastructure, deployed for blocking and prevention. For example, using firewall logs for intrusion detection – often explained and justified, but not as commonly used – is an example of that.

What is even more interesting is that logging and monitoring controls (notice that these are considered security “controls” even though they don't really “control” anything) allow one to measure security and compliance by building metrics and trends. This means that one can use such data for a range of applications, from a simple “Top users by bandwidth” report obtained from firewall logs, all the way to sophisticated tracking of card data movements and use.

That is exactly why most regulations, industry mandates, and even “best practices” frameworks (International Organization for Standardization [ISO], COBIT, ITIL) call for logging and monitoring. Of course, PCI DSS is not an exception!

Last, but not least, if the worst does happen, then you would need to have as much data as possible during your incident response process: you might not use it all, but having a reliable log, assessment trail, and network capture data from all affected systems is indispensable for a hectic postincident environment.

Requirements 10 and 11 are easily capable of inflating PCI compliance costs to the point of consuming the small margins of card transactions. No one wants to lose money to be PCI compliant. Therefore, the ability to meet the requirements above all must make business sense. Nowhere else in PCI compliance does the middle ground of design philosophy come into play more than in the discipline of monitoring, but this is also where minimizing the risk can hurt most.

Finally, assuming that you've designed your PCI environment to have appropriate physical and logical boundaries through the use of segregated networks and dedicated application space as we described in Chapter 4, “Building and Maintaining a Secure Network,” you should be able to identify the boundaries of your monitoring scope. If you haven't done this part, go back to Requirement 1 and start over!

System and network logs, one of the key sources of monitoring data, are often called the “untapped riches.” The now-famous humorous security calendar proclaims “Logfiles: The Data Center Equivalent of Compost: Let'em Rot.”[1] Others are not as loud about it and quietly choose to follow this maxim and ignore logs – at their own peril – altogether. Many organizations, whether under PCI DSS or not, still think of security as blocking and denying and leave monitoring and logging aside, despite their importance.

On the other hand, as computer and Internet technology continues to spread and computers start playing an even more important role in our lives, the records that they produce, such as logs and other traces, start to play a bigger role. From firewalls and routers to databases and enterprise applications, to wireless access points and Voice over Internet Protocol (VoIP) gateways, logs are being spewed forth at an ever-increasing pace. Both security and other information technology (IT) components not only increase in numbers but also often come with more logging enabled out of the box. An example of this trend includes Linux operating system as well as Web servers, both commercial and open-source, that now ship with increased levels of logging out of the box. All those systems, both legacy and modern, are known to generate copious amounts of logs, assessment trails, records, and alerts. In addition, such additional monitoring methods as network packet capture, database activity monitoring (DAM), and special-purpose application monitoring are becoming common as well. And all this data begs for constant attention!

But this is easier said than done. Immense volumes of monitoring data are being generated on payment card processing networks. In turn, it results in a need to manage, store, and search all this data. Moreover, such review of data needs to happen both reactively – after a suspected incident – and proactively – in search of potential risks and future problems. For example, a typical large retailer generates hundreds of thousands of log messages per day amounting to many terabytes per year. An online merchant can generate millions of various log messages every day. One of America's largest retailers has more than 60 TB of log data on their systems at any given time. Unlike other companies, retailers do not have the option of not using logging due to PCI DSS.

To start our discussion of PCI logging and monitoring requirements, Table 9.1 shows a sample list of technologies that produce logs of relevance to PCI. Though this list is not comprehensive, it is likely that the readers find at least one system that they have in their cardholder data environment and for which logs are not being collected, much less looked at, and that is not being monitored at all.

In addition, Table 9.2 shows a few of the technologies not commonly monitored using logs.

Many companies and government agencies are trying to set up repeatable log collection, centralization, and analysis processes and tools.

Despite the multitude of log sources and types, people typically start from network and firewall logs and then progress upward on the protocol stack as well as sideways toward other nonnetwork applications. For example, just about any firewall or network administrator will look at a simple summary of connections that his or her Cisco ASA (or older PIX) or Checkpoint firewall is logging. Many firewalls log in standard syslog format, and such logs are easy to collect and review.

For example, here is a Juniper firewall log message in syslog format:

And, here is the one from Cisco ASA firewall device:

Reviewing network intrusion detection system (NIDS) or IPS logs, although “interesting” in case of an incident, is often a very frustrating task since NIDS would sometimes produce “false alarms” and dutifully log them. Still, NIDS log analysis, at least the postmortem kind for investigative purposes, often happens right after firewall logs are looked at.

As a result, organizations deploy their log management infrastructure for compliance, security, or operational uses; the value of such information for security is undeniable, and logs can, in most cases, be easily centralized for analysis.

Even though system administrators always knew to look at logs in case of problems, massive server operating system (both Windows and Unix/Linux variants) log analysis didn't materialize until more recently. Collecting logs from Windows servers, for example, was hindered by the lack of agentless log collection tools, such as Lasso, that only emerged in the last year or two. On the other hand, Unix server log analysis was severely undercut by a total lack of unified format for log content in syslog records.

Web server logs were long analyzed by marketing departments to check on their online campaign successes. Most Web server administrators would also not ignore those logs. However, because Web servers don't have native log forwarding capabilities (most log to files stored on the server itself), consistent centralized Web log analysis for both security and other IT purposes is still ramping up.

For example, the open-source Apache Web server has several types of logs. The most typical among them are access_log that contains all page requests made to the server (with their response codes) and error_log that contains various errors and problems. Other Apache logs relate to Secure Sockets Layer (SSL) (ssl_error_log) as well as optional granular assessment logs that can be configured using tools such as ModSecurity (which produces an additional highly-detailed audit_log).

Similarly, e-mail tracking through e-mail server logs languishes in a somewhat similar manner: people only turn to e-mail logs when something goes wrong (e-mail failures) or horribly wrong (external party subpoenas your logs). Lack of native centralization and, to some extent, complicated log formats slowed down the e-mail log analysis initiatives.

Even more than e-mail, database logging wasn't on the radar of most IT folks until last year. In fact, IT folks were perfectly happy with the fact that even though Relational Database Management Systems (RDBMSs) had extensive logging and data access assessment capabilities, most of them were never turned on – many times citing performance issues. Oracle, Microsoft Structured Query Language (SQL) Server, IBM DB2, and MySQL all provide excellent logging, if you know how to enable it, configure it for your specific needs, and analyze and leverage the resulting onslaught of data. In the context of PCI DSS, database monitoring is often performed not using logs but instead using a separate software called DAM.

What's next? Web applications and large enterprise application frameworks largely lived in a world of their own, but now people are starting to realize that their log data provides unique insight into insider attacks, insider data theft, and other trusted access abuse. Additionally, desktop operating system log analysis from large numbers of deployed desktops will also follow.

Before we begin with covering additional details on logging and monitoring in PCI, one question needs to be addressed. It often happens that PCI Qualified Security Assessors (QSAs) or security consultants are approached by the merchants: what exactly must they log and monitor for PCI DSS compliance? The honest answer to the above question is that there is no list of what exactly you must be logging due to PCI or, pretty much, any other recent compliance mandate. That is true despite the fact that PCI rules are more specific than most other recent regulations, affecting information security. However, the above does not mean that you can log nothing.

The only thing that can be explained is what you should be logging. There is no easy “MUST-log-this” list; it is pretty much up to individual assessor, consultant, vendor, engineer, and so forth to interpret – not simply “read,” but interpret! – PCI DSS guidance in your own environment. In addition, when planning what to log and monitor, it makes sense to start from compliance requirement as opposed to end with what PCI DSS suggests. After all, organization can derive value from using it, even without regulatory or industry compliance.

So, which logs are relevant to your PCI project? In some circumstances, the answer is “all of them,” but it is more likely that logs from systems that handle credit-card information, as well as systems they connect to, will be in-scope. Please refer to the data flow diagram that was described in Chapter 4, “Building and Maintaining a Secure Network,” to determine which systems actually PCI DSS requirements apply to all members, merchants, and service providers that store, process, or transmit cardholder data. Additionally, these requirements apply to all “system components,” which are defined as “any network component, server, or application included in, or connected to, the cardholder data environment.” Network components include, but are not limited to, firewalls, switches, routers, wireless access points, network appliances, and other security appliances. Servers include, but are not limited to, Web, database, authentication, Domain Name System (DNS), e-mail, proxy, and Network Time Protocol (NTP). Applications include all off-the-shelf and custom-built applications, including internally facing and externally facing Web applications.

By the way, it is important to remind you that approaching logging and monitoring with the sole purpose of becoming PCI compliant is not only wasteful but can actually undermine the intent of PCI DSS compliance. Staring from its intent – cardholder data security – is the way to go, which we advocate.

The following are a few common uses for log information, besides PCI DSS compliance:

Let's quickly go through Requirement 10, which directly addresses logging. We will go through it line by line and then go into details, examples, and implementation guidance.

The requirement itself is called “Track and monitor all access to network resources and cardholder data” and is organized under the “Regularly monitor and test networks” heading. The theme thus deals with both periodic (test) and ongoing (monitor) aspects of maintaining your security; we are focusing on logging and monitoring and have addressed periodic testing in Chapter 8, “Vulnerability Management.” More specifically, it requires a network operator to track and monitor all access to network resources and cardholder data. Thus, both network resources that handle the data and the data itself are subject to those protections.

Further, the requirement states that logging is critical, primarily when “something does go wrong” and one needs to “determine the cause of a compromise” or other problem. Indeed, logs are of immense importance for incident response. However, using logs for routine user tracking and system analysis cannot be underestimated. Next, the requirement is organized in several sections on process, events that need to be logged, suggested level of details, time synchronization, assessment log security, required log review, and log retention policy.

Specifically, Requirement 10.1 covers “establish[ing] a process for linking all access to system components (especially access done with administrative privileges such as root) to each individual user.” This is a very interesting requirement indeed; it doesn't just mandate for logs to be there or for a logging process to be set, but instead it mentions that logs must be tied to individual persons (not computers or “devices” where they are produced). It is this requirement that often creates problems for PCI implementers because many think of logs as “records of people actions,” while in reality they will only have the “records of computer actions.” Mapping the latter to actual flesh-and-blood users often presents an additional challenge. By the way, PCI DSS Requirement 8.1 mandates that an organization “assigns all users a unique ID before allowing them to access system components or cardholder data, which” helps to make the logs more useful here.

Next, Section 10.2 defines a minimum list of system events to be logged (or, to allow “the events to be reconstructed”). Such requirements are motivated by the need to assess and monitor user actions as well as other events that can affect credit-card data (such as system failures).

Following is the list from the requirements (events that must be logged) from PCI DSS:

As can be seen, this covers data access, privileged user actions, log access and initialization, failed and invalid access attempts, authentication and authorization decisions, and system object changes. It is important to note that such a list has its roots in IT governance “best practices,” which prescribe monitoring access, authentication, authorization change management, system availability, and suspicious activity. Thus, other regulations, such as the Sarbanes–Oxley Act and IT governance frameworks such as COBIT, have very similar lists of events that need to be logged.

Table 9.3 is a practical example of the list on the previous page.

Moreover, PCI DSS Requirement 10 goes into an even deeper level of detail and covers specific data fields or values that need to be logged for each event. They provide a healthy minimum requirement, which is commonly exceeded by logging mechanisms in various IT platforms.

Such fields are as follows:

As shown, this minimum list contains all the basic attributes needed for incident analysis and for answering the questions: when, who, where, what, and where from. For example, if you are trying to discover who modified a credit-card database to copy all the transactions with all the details into a hidden file (a typical insider privilege abuse), you would need-to-know all the above. Table 9.4 summarizes the above fields in this case.

Requirement 10.4 addresses a commonly overlooked but critical requirement: a need to have accurate and consistent time in all of the logs. It seems fairly straightforward that time and security event monitoring would go hand in hand as well.

System time is frequently found to be arbitrary in a home or small office network. It's whatever time your server was set at, or if you designed your network for some level of reliance, your systems are configured to obtain time synchronization from a reliable source, like the NTP servers.

A need to “synchronize all critical system clocks and times” can make or break your incident response or lead to countless hours spent figuring out the actual times of events by correlating multiple sources of information together. In some cases, uncertainty about the log timestamps might even lead a court case to be dismissed because uncertainly about timestamps might lead to uncertainty in other claims as well. For example, from “so you are saying you are not sure when exactly it happened?” an expert attorney might jump to “so maybe you are not even sure what happened?” Fortunately, this requirement is relatively straightforward to address by configuring an NTP environment and then configuring all servers to synchronize time with it. The primary NTP servers can synchronize time with time.nist.gov or other official time sources, also called “Stratum 1” sources.

Stratum 1 time sources are those devices acquiring time data from direct sources like the atomic clocks run by various government entities or Global Positioning System (GPS) satellites. Local hardware, in fact, is considered Stratum 1; it gets time from its own CMOS. Stratum 2 gets its time from Stratum 1 and so on. For purposes of PCI compliance, Stratum 2 is sufficient to “prove” time, as long as all systems in the PCI environment synchronize their clocks with the Stratum 2 source via the network. Obviously, having reliable power and network is critical to this sort of approach.

Security of the logs themselves is of paramount importance for reasons similar to the above concerns about the log time synchronization. Requirement 10.5 states that one needs to “secure audit trails so they cannot be altered” and then clarifies various risks that need to be addressed.

Because it is a key issue, we will look at the security of monitoring data and logs in the next section.

Although PCI is more about being compliant than about being hacked (well, not directly!), logs certainly help to answer this question. However, if logs themselves are compromised by the attackers and the log server is broken into, they lose all values for either security or compliance purposes. Thus, having assured log confidentiality, integrity, and availability (CIA) is a requirement for PCI as well as a best practice for other log uses.

First, one needs to address all the CIA of logs. Section 10.5.1 of PCI DSS covers the confidentiality: “Limit viewing of audit trails to those with a job-related need.” This means that only those who need to see the logs to accomplish their jobs should be able to. What is so sensitive about logs? One of the obvious answers is that authentication-related logs will always contain usernames. Although not truly secret, username information provides 50 percent of the information needed for password guessing (password being the other 50 percent). Why give possible attackers (whether internal or external) this information? Moreover, because of users mistyping their credentials, it is not uncommon for passwords themselves to show up in logs. Poorly written Web applications might result in a password being logged together with the Web Uniform Resource Locator (URL) in Web server logs. Similarly, a Unix server log might contain a user password if the user accidentally presses “Enter” one extra time while logging in.

Next comes “integrity.” As per Section 10.5.2 of PCI DSS, one needs to “protect audit trail files from unauthorized modifications.” This one is blatantly obvious; because if logs can be modified by unauthorized parties (or by anybody), they stop being an objective assessment trail of system and user activities.

However, one needs to preserve the logs not only from malicious users but also from system failures and consequences of system configuration errors. This touches upon both the “availability” and “integrity” of log data. Specifically, Section 10.5.3 of PCI DSS covers that one needs to “promptly back-up audit trail files to a centralized log server or media that is difficult to alter.” Indeed, centralizing logs to a server or a set of servers that can be used for log analysis is essential for both log protection and increasing log usefulness. Backing up logs to CDs or DVDs (or tapes, for that matter) is another action one has to perform as a result of this requirement. One should always keep in mind that logs on tape are not easily accessible and not searchable in case of an incident.

Many pieces of network infrastructure such as routers and switches are designed to log to an external server and only preserve a minimum (or none) of logs on the device itself. Thus, for those systems, centralizing logs is most critical. Requirement 10.5.4 of PCI DSS states the need to “copy logs for wireless networks onto a log server on the internal LAN.”

To further decrease the risk of log alteration as well as to enable proof that such alteration didn't take place, Requirement 10.5.5 calls for the “use file integrity monitoring and change detection software on logs to ensure that existing log data cannot be changed without generating alerts.” At the same time, adding new log data to a log file should not generate an alert because log files tend to grow and not shrink on their own (unless logs are rotated or archived to external storage). File integrity monitoring systems use cryptographic hashing algorithms to compare files to a known good copy. The issue with logs is that log files tend to grow due to new record addition, thus undermining the utility of integrity checking. To resolve this difficulty one should note that integrity monitoring can only assure the integrity of logs that are not being actively written to by the logging components. However, there are solutions that can verify the integrity of growing logs.

The next requirement is one of the most important as well as one of the most overlooked. Many PCI implementers simply forget that PCI Requirement 10 does not just call for “having logs” but also for “having the logs and looking at them.” Specifically, Section 10.6 states that the PCI organization must, as per PCI DSS, “review logs for all system components at least daily. Log reviews must include those servers that perform security functions like IDSs and AAA servers (e.g., RADIUS).”

Thus, the requirement covers the scope of log sources that need to be “reviewed daily” and not just configured to log and have logs preserved or centralized. Given that a Fortune 1000 IT environment might produce gigabytes of logs per day, it is humanly impossible to read all of the logs. That is why a note is added to this requirement of PCI DSS that states that “Log harvesting, parsing, and alerting tools may be used to achieve compliance with Requirement 10.6.” Indeed, log management tools are the only way to go to satisfy this requirement.

The final Requirement 10.7 deals with another hugely important logging question – log retention. It says to “retain audit trail history for at least one year, with a minimum of three months online availability.” Unlike countless other requirements, this deals with the complicated log retention question directly. Thus, if you are not able to go back 1 year and look at the logs, you are in violation. Moreover, PCI DSS in its updated version v1.1 got more prescriptive when a 1-year requirement was added explicitly.

So, let us summarize what we have learned so far on logging in PCI:

Now, we are ready to dig deeper to discover that logs and monitoring “live” not only within Requirement 10 but in all other PCI requirements.

Although many people think that logs in PCI are represented only by Requirement 10, the reality is more complicated: logs are in fact present, undercover, in all other sections. We will now reveal where they hide in other sections. Table 9.5 highlights some of the places where logging requirements are implied or mentioned. The overall theme here is that logging and log management assists with validation and verification of many other requirements.

Now, let's dive deeper into the role of logs to further explain that logs are not only about the Requirement 10. Just about every claim that is made to satisfy the requirements, such as data encryption or antivirus updates, can make effective use of log files to actually substantiate it.

For example, Requirement 1, “Install and maintain a firewall configuration to protect cardholder data,” mentions that organizations must have “a formal process for approving and testing all external network connections and changes to the firewall configuration.” However, after such a process is established, one needs to validate that firewall configuration changes do happen with authorization and in accordance with documented change management procedures. That is where logging becomes extremely handy, because it shows you what actually happened and not just what was supposed to happen.

Specifically, seeing a message such as this Cisco ASA appliance record should indicate that someone is likely trying to modify the appliance configuration.

Or this message indicates a failure to complete configuration update:

Other log-related areas within Requirement 1 include Section 1.1.6.

“Justification and documentation for any available protocols besides Hypertext Transfer Protocol (HTTP), SSL, Secure Shell (SSH), and virtual private network (VPN) where logs should be used to watch for all events triggered due to such communication.”

Section 1.1.7: Justification and documentation for any risky protocols allowed (for example, File Transfer Protocol [FTP]), which includes the reason for use of protocol and security features implemented, where logs help to catalog the user of “risky” protocols and then monitor such use.

The entire Requirement 1.3 contains guidance to firewall configuration, with specific statements about inbound and outbound connectivity. One must use firewall logs to verify this; even a review of configuration would not be sufficient, because only logs show “how it really happened” and not just “how it was configured.”

Similarly, Requirement 2 talks about password management “best practices” as well as general security hardening, such as not running unneeded services. Logs can show when such previously disabled services are being started, either by misinformed system administrators or by attackers.

For example, if Apache Web server is disabled on an e-mail server system, a message such as the following should trigger an alert because the service should not be starting or restarting.

Further, Requirement 3, which deals with data encryption, has direct and unambiguous links to logging. For example, the entire subsection 3.6, shown below in an abbreviated form, implies having logs to verify that such activity actually takes place.

Specifically, key generation, distribution, and revocation are logged by most encryption systems, and such logs are critical for satisfying this requirement.

Requirement 4, which also deals with encryption, has logging implications for similar reasons.

Requirement 5 refers to antivirus defenses. Of course, to satisfy Section 5.2, which requires that you “Ensure that all antivirus mechanisms are current, actively running, and capable of generating audit logs,” one needs to see such mentioned logs.

For example, Symantec AntiVirus might produce the following log record that occurs when the antivirus software experiences problems and cannot continue scanning, thus putting you in violation of PCI DSS rules.

Thus, unless you are monitoring for such event records, you cannot be in compliance.

So, even the requirement to “use and regularly update antivirus software” will likely generate requests for log data during the assessment, because the information is present in antivirus assessment logs. It is also well-known that failed antivirus updates, also reflected in logs, expose the company to malware risks because antivirus without the latest signature updates only creates a false sense of security and undermines the compliance effort.

Requirement 6 is in the same league: it calls for the organizations to “Develop and maintain secure systems and applications,” which is unthinkable without a strong assessment logging function and application security monitoring.

Requirement 7, which states that one needs to “Restrict access to cardholder data by business need-to-know,” requires logs to validate who actually had access to said data. If the users who should be prevented from seeing the data appear in the log files as accessing the data usefully, remediation is needed.

Assigning an unique ID to each user accessing the system fits with other security “best practices.” In PCI, it is not just a “best practice”; it is a requirement (Requirement 8 “Assign a unique ID to each person with computer access”).

Obviously, one needs to “Control addition, deletion, and modification of user IDs, credentials, and other identifier objects” (Section 8.5.1 of PCI DSS). Most systems log such activities.

For example, the message below indicates a new user being added to a PIX/ASA firewall.

In addition, Section 8.5.9, “Change user passwords at least every 90 days,” can also be verified by reviewing the logs files from the server to assure that all the accounts have their password changed at least every 90 days.

Requirement 9 presents a new realm of security – physical access control. Even Section 9.4 that covers maintaining a visitor log (likely in the form of a physical log book) is connected to log management if such a visitor log is electronic. There are separate data retention requirements for such logs: “Use a visitor log to maintain a physical assessment trail of visitor activity. Retain this log for a minimum of three months, unless otherwise restricted by law.”

Requirement 11 addresses the need to scan (or “test”) the in-scope systems for vulnerabilities. However, it also calls for the use of IDS or IPS in Section 11.4: “Use network intrusion detection systems, host-based intrusion detection systems, and intrusion prevention systems to monitor all network traffic and alert personnel to suspected compromises. Keep all intrusion detection and prevention engines up-to-date.” Intrusion detection is only useful if monitored.

Requirement 12 covers the issues on a higher level – security policy as well as security standards and daily operational procedures (e.g., a procedure for daily log review mandates by Requirement 10 should be reflected here). However, it also has logging implications because assessment logging should be a part of every security policy. In addition, incident response requirements are also tied to logging: “Establish, document, and distribute security incident response and escalation procedures to ensure timely and effective handling of all situations” is unthinkable to satisfy without effective collection and timely review of log data.

Thus, event logging and security monitoring in PCI DSS program goes much beyond Requirement 10. Only through careful data collection and analysis can companies meet the broad requirements of PCI.

At this stage, we went through all of the PCI guidelines and uncovered where logging and monitoring are referenced. We now have a mammoth task ahead – how to address all those requirements? Let's quickly go back and review what we learned, both in Requirement 10 and beyond.

Let's analyze the above requirements and needs to determine what kind of tools we might need to develop or procure.

First, let's note that if we are talking about a single server and a single piece of network gear such as a router, there might be no need for automation and tools. One can easily configure logging and then look at the logs (measuring a few pages of text a day or more, in case more comprehensive assessing is performed), as well as save a copy of said logs to make sure that one can go back a year and review the old logs if needed. However, this approach fails miserably and dramatically when the number of systems grows from 1 to, say, 10. In reality, a large e-commerce site or a whole chain of stores might easily have thousands of in-scope systems, starting from mainframes with customer databases down to servers to complex network architectures (including classic LANs, WANs, wireless networks, and remote access systems with hundreds of remote users) to point of sale (POS) systems and all the way down to wireless card scanners. A handheld wireless card scanner is “a credit-card processing system” and thus is in-scope for PCI compliance. Looking through recent media headlines, such as credit card compromises at BestBuy and other “brick and mortar” retailers, one learns that credit-card information has indeed been stolen this way.

Once it has been accepted that manual review of logs is not feasible or effective, the next attempt to satisfy the requirements usually comes in the form of scripts written by system administrators, to filter, review, and centralize the logs as well as makeshift remote logging collectors (usually limited to those log sources that support syslog, which is easy to forward to another system).

For example, one might configure all in-scope Unix servers to log a single “log server” and then write a Perl script that will scan the logs for specific strings (such as “fail*,” “attack,” “denied,” “modify,” and so forth) and then send an e-mail to the administrator upon finding one of those strings. Another script might be used to summarize the log records into a simple “report” that highlights, for example, “Top Users” or “Top IP Addresses.” A few creative tools were written to implement various advanced methods of log analysis, such as rule-based or stateful correlation.

Such “solutions” work well and do not require any initial investment. Other advantages of such “homegrown” approaches are as follows:

What makes it even easier is the availability of open source and freeware tools to address some of the pieces of log management for PCI. For example, Table 9.6 summarizes a few popular tools that can be (and in fact, have been) used in PCI log management projects.

However, if an author's extensive experience with logging and monitoring is any indication, most if not all projects of this type, no matter how well thought-out and no matter how well funded, will fail.

One reason for it is the scalability of such custom tool. It might work great in a laboratory but fall completely flat on its face in a production environment due to data volume, complexities of infrastructure, and so forth. Log analysis or security monitoring system needs to be capable to handle volume, not only live flow but also storage. A lot of log tools work well on 10 MB of logs, but then again, so does a human brain. When you move to terabyte volumes, a lot of simple things start to require engineering marvels. Is it as hard as getting the Linux kernel (the pinnacle of open-source engineering) to achieve high performance? Probably not as hard, but the logging tool developer needs to be both a log expert and a performance expert.

In addition, the question is also whether this tool will scale with your organization or will it require a complete redesign and then rewrite when your environment grows and/or your needs change? Such efforts are usually a big drain on IT teams because people who might be doing things critical to maintaining a well-oiled “IT machine,” all start writing code instead – and often not the most secure and efficient code at that.

Next, management often likes to point out that such an approach doesn't pass “the bus test” (namely, there is no satisfying and credible answer to the question, “What do we do if the smart guy who wrote this wonder of log analysis technology gets run over by a bus?”).

And the final, most terrifying reason: ongoing maintenance of such tools is what deals a mortal blow to many in-house log analysis projects. System vendors change, log formats change, often without notice and without documents (provided they did have documents in the first place) thus leading to log analysis system failures with subsequent gaps in PCI-compliance log review or, worse, log collection or retention. We are aware of more than one case where a large corporation abandoned a well-built in-house log analysis tool (with capabilities superior to those of commercial vendors at the time) after spending literally millions of dollars for that reason alone: efforts to update the tool levied a heavy “tax” on team's productivity.

Thus, many people turn to commercial vendors when looking for solutions to PCI logging and monitoring challenges. On the logging side, commercial log management solutions can aggregate all data from the in-scope entities, whether applications, servers, or network gear. Such solutions enable satisfying the log data collection, monitoring, analysis, data protection, and data retention. (Why do we keep saying “retention” where some people would have used to term “storage?” It is important to note that “retention” usually implies making sure that data is stored.) Vendors also help with system configuration guidance to enable optimum logging (sometimes for a fee as “professional services”). Advantages of such an approach are obvious: on day 1, you get a supported solution as well as a degree of certainty that the vendor will maintain and improve the technology as well as have a roadmap for addressing other log-related organization needs beyond PCI compliance.

On the negative side of acquiring a PCI logging solution from a vendor sits a landmine of “unmet requirements.” It might happen that what was bought and deployed doesn't match what was imagined and needed. Many factors contribute to such a situation: starting from aggressive vendor sales tactics (“overselling”) to insufficient onsite testing and to not thinking about the needs before talking to vendors. Without a doubt, if you “don't know what you need,” it is unlikely that you'd buy “exactly what you need.”

Fortunately, there are simple things you can do to avoid the pitfall of unmet requirements when acquiring a log management solution.

Look through various commercial log management such as LogLogic (www.loglogic.com), ArcSight (www.arcsight.com), Splunk (www.splunk.com), or others to “case the joint” and to see what is out there. Congratulations! You are ready to talk to vendors. Here are a few additional questions to ask the vendor:

Let's take a more detailed look at the capabilities of a typical log management solution, a common choice for security monitoring for PCI DSS. The first thing to mention is that such solutions make logs useful for security, IT performance monitoring, system and network troubleshooting, investigations, assessment, and, obviously, compliance such as PCI DSS.

For example, real-time alerting is useful primarily as a threat detection measure. To provide such data protection measures, companies should implement a log management solution that enables administrators to set alerts on all applications, devices, and system logs. This enables them to provide evidence that the infrastructure has been configured properly and that misconfigured or vulnerable systems are not providing a backdoor for intruders or malicious insiders. Alerts can provide administrators with early warning of misuse and attacks, allowing them to isolate and fix the problem before damage occurs or data is lost, and of various data access policies and processes not being followed.

Securing the CIA of log data is explicitly mentioned in the PCI requirements above; thus, it is crucial to any implementation of a log management tool. This not only serves to reduce the risk of this vital information leaking by means of logs (confidentiality) and prevents it from being altered or lost thereby reducing its relevance, immutability, and forensic quality (integrity) but also makes sure that log data is there when you need it (availability). It is hardly possible to say which is the most important and thus the CIA triad lives on.

A need to be able to use logs to address all those PCI requirements brings us to access management and change management. Although two different areas with little overlap, both access and change managements are critical to meeting PCI compliance requirements as well as other regulations and IT governance frameworks, such as ITIL, COBIT, or various ISO guidance documents. Strong access and change control measures ensure that only authorized users can access or take action on critical data. The PCI standard mandates that companies maintain a complete record of access (both failed and successful), activity, and configuration changes for applications, servers, and network devices. Logs are that record. Thus, such log data allows IT to set up alerts to unusual or suspicious network behavior and provide information to assessors with complete and accurate validation of security policy enforcement and segregation of duties.

Further, a log management solution allows administrators to monitor who has permission to access or make changes to devices and applications in the network. It also enables administrators to create a complete assessment trail across devices and protect network resources from unauthorized access or modifications. An effective log management tool supports centralized, automated storage of collected data for faster, more reliable data retrieval during an assessment or while investigating suspicious behavior.

You probably remember that PCI compliance necessitates ongoing monitoring of network activity (see Requirement 11 and others) to validate that processes and policies for security, change and access management, and user validation are in place and up-to-date.

Logging and monitoring allow for fast problem isolation and thorough analysis when something goes (reactive analysis) or is about to go wrong (proactive analysis). Ongoing and automated log monitoring gives administrators greater insight into the PCI environment at all times so that unusual user activity, unauthorized access, or even risky insider behavior can be identified – and stopped – immediately.

In light of the above, the components of an effective log management solution are as follows:

It is critical to remember that the scope of security monitoring in PCI DSS is not limited to logs because system logs alone do not cover all the monitoring needs. Additional monitoring requirements are covered by the following technologies (as well as process requirements that accompany them):

We will cover the critical issues related to these technologies in the section below.

NIDS and IPSs are becoming a standard information security safeguard. Together with firewalls and vulnerability scanners, intrusion detection is one of the pillars of modern computer security. When referring to IDSs and IPSs today, security professional typically refers to NIDS and network IPS. As we covered in Chapter 4, “Building and Maintaining a Secure Network,” the former sniffs the network, looking for traces of attacks, whereas the latter sits “inline” and passes (or blocks) network traffic.

NIDS monitors the entire subnet for network attacks against machines connected to it, using a database of attack signatures or a set of algorithms to detect anomalies in network traffic. See Fig. 9.1 for a typical NIDS deployment scenario.

On the other hand, network IPS sits at a network choke point and protects such a network of systems from inbound attacks. To simplify the difference, IDS alerts whereas IPS blocks. See Fig. 9.2 for a typical network IPS deployment.

The core technology of picking “badness” from the network traffic with a subsequent alert (IDS) or blocking (IPS) is essentially similar. Even when intrusion prevention functionality is integrated with other functions to be deployed as so-called Unified Threat Management (UTM), the idea remains the same: network traffic passes through the device with malicious traffic stopped, suspicious traffic logged, and benign passed through.

Also important is the fact that most of today's IDS and IPS rely upon the knowledge of attacks and thus require ongoing updates of signatures, rules, attack traces to look for, and so forth. This is exactly why PCI DSS mandates that IDS and IPS are not only deployed but also frequently updated and managed in accordance with manufacturer's recommendations.

In the context of PCI, IDS and IPS technologies are mentioned in the context of monitoring. Even though IDS can only alert and log attacks while IPS adds blocking functionality, both can and must be used to notify the security personnel about malicious and suspicious activities on the cardholder data networks. Below we present a few useful tips for deploying IDS and IPS for PCI DSS compliance and card data security.

Despite the domination of commercial vendors, the free open-source IDS/IPS Snort, now developed and maintained by its corporate parent, Sourcefire, is likely the best IDS/IPS by the number of deployments worldwide. Given its price (free) and reliable rule updates from Sourcefire (www.sourcefire.com), it makes a logical first choice for smaller organizations; it shouldn't be taken off the shortlist even for larger organizations seeking to implement intrusion detection or prevention.

Although a detailed review of IDS and IPS technologies and practices goes well beyond the scope of this book, we would like to present a few key practices for making your PCI-driven deployment successful.

First, four key facts about IDS and IPS, which are also highlighted in the PCI standard:

The above four facts define how IDS and IPS are used for the cardholder requirement. Despite the above knowledge, IDS technologies are not the easiest one to deploy, especially in light of the number 3 consideration above. PCI DSS-driven IDS deployments suffer from a few of the common mistakes covered below.

First, using an IDS or an IPS to protect the cardholder environment and to satisfy PCI DSS requirement is impossible without giving it an ability to see all the network traffic. In other words, deploying an NIDS without sufficient network environment planning is a big mistake that reduces, if not destroys, the value of such tools. Network IPS, for example, should be deployed on the network choke point such as right inside the firewall leading to cardholder network, on the appropriate internal network segment, or in the De-Militarized Zone (DMZ). For the shared Ethernet-based networks, IDS will see all the network traffic within the Ethernet collision domain or subnet and also destined to and from the subnet but no more. For the switched networks, there are several IDS deployment scenarios that use special switch capabilities such as port mirroring or spanning. When one or more IDS devices are deployed, it is your responsibility to confirm that they can “cover” the entire “in-scope” network.

Second, even if an IDS is deployed appropriately, but nobody is looking at the alerts it generates, the deployment will end in failure and will not lead to PCI compliance. It's well known that IDS is a “detection” technology, and it never promised to be a “shoot-and-forget” means of thwarting attacks. Although in some cases, the organization might get away with dropping the firewall in place and configuring the policy, such a deployment scenario never works for intrusion detection. If IDS alerts are reviewed only after a successful compromise, the system turns into an overpriced incident response helper tool, clearly not what the technology designers had in mind. Even with IPS, a lot of suspicious indicators are not reliable enough to be blocked automatically, thus monitoring is just as critical as with IDS.

PCI DSS Requirement 12.5.2 does state that an organization needs to “Monitor and analyze security alerts and information, and distribute to appropriate personnel.” Still, despite this, many organizations deploy IDS and develop a no response policy. As a result, their network IPS is deployed, it “sees” all the traffic, and there is somebody reviewing the alert stream. But what is the response for each potential alert? Panic, maybe? Does the person viewing the alerts know the best course of action needed for each event? What alerts are typically “false positives” – alerts being triggered on benign activity – and “false alarms” – alerts being triggered on attacks that cannot harm the target systems – in the protected environment? Unless these questions are answered, it is likely that no intelligent action is being taken based on IDS alerts – a big mistake by itself, even without PCI DSS.

The fourth and final mistake is simply not accepting the inherent limitations of network intrusion protection technology. Although anomaly-based IDSs might detect an unknown attack, most signature-based IDS will miss a new exploit if there is no rule written for it. IDS and IPS must frequently receive vendor signature updates, as mandates by the PCI DSS. Even if updates are applied on a schedule, exploits that are unknown to the IDS vendor will probably not be caught by the signature-based system. Attackers may also try to blind or evade the NIDS by using many tools available for download. There is a constant battle between the technology developers and those who want to escape detection. IPS/IDS are becoming more sophisticated and able to see through the old evasion methods, but new approaches are constantly being developed by attackers. Those deploying the NIDS technology should be aware of its limitations and practice “defense-in-depth” by deploying multiple and diverse security solutions.

Thus, IDS/IPS is a key monitoring technology for PCI DSS and data protection; however, when deploying it, many pitfalls need to be considered if it were to be useful for PCI compliance and security.

In the prehistoric days of security industry, before all the current compliance frenzy and way before PCI DSS, the idea of monitoring key system files for changes was invented. As early system administrators engaged in an unequal battle with hackers who compromised almost every server connected to the Internet, the idea of a quick and easy way for verifying that system files was not modified and thus not subverted by hackers seemed like a god-send.

Indeed, just run a quick “scan” of a filesystem, compute cryptographic checksums, save them in a safe place (a floppy comes to mind – we are talking 90s here, after all), and then have a “known good” record of the system files. In case of problems, run another checksum computation and compare the results; if differences are found, then something or someone has subverted the files.

That is exactly the idea of integrity checking and monitoring. The tools such as the pioneering tripwire (now developed by its corporate parent, Tripwire, Inc) have matured significantly and offer near real-time checks, an ability to revert to the previous version of the changed files, as well as a detailed analysis of observed changes across a wide range of platforms for servers, desktops, and even network devices.

As we mentioned, PCI DSS mandates the use of such tools via Requirement 11.5. Namely, “Deploy file-integrity monitoring software to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly.”

This means that the tools must be deployed on in-scope systems, key files need to be monitored, and comparisons are to be run at least weekly. Indeed, knowing that the server has been compromised by attackers a week after the incident is a lot better than what some recent credit-card losses indicate. For example, TJX card theft was discovered years after the actual intrusion took place, causing the company some massive embarrassment.

On the tools side, although Tripwire rules the commercial file integrity monitoring, its open-source clone is also of the best known free, open-source tools. Free tripwire is even included with some Linux and Unix operating systems.

Some of the challenges with such tools include creating a list of key files to checksum. Typically, relying on integrity checking tool vendor default (or even “PCI DSS-focused”) policy is a good idea. Here is an example list of such files for a typical Linux server:

Also, just as with IDS and IPS, a response policy in case of breach of integrity is essential.

Finally, let's review our suggestions on monitoring and logging policy for PCI DSS.

By now the reader should be convinced that it is impossible to comply with PCI requirements without log data management processes and technologies in place. Complete log data is needed to prove that security, change management, access control, and other required processes and policies are in use, up-to-date, and are being adhered to. In addition, when managed well, log data can protect companies when compliance-related legal issues arise (e.g., when processes and procedures are in question or when an e-discovery process is initiated as part of an ongoing investigation). Not only does log data enable compliance but also allows companies to prove that they are implementing and continuously monitoring the processes outlined by the requirements.

Additionally, we will present a few common mistakes noted in PCI DSS-driven logging implementations.

Next, we present two of the case studies that illustrate what is covered in this chapter.

In conclusion, the authors would like to stress a few points that were covered as well as leave readers with a few thoughts about how PCI logging fits into the bigger picture of IT governance and risk management. Despite the fact that this book is about PCI DSS, we do not recommend that you embark on a log management project or on a security monitoring project solely for PCI DSS. Taking control of your logs is useful for a huge number of IT and information security goals; thus doing logs “just for compliance” is akin to using an expensive and powerful tool as a hammer. Overall, a common scenario observed by the authors is “buy for compliance [such as PCI], use for all the needed purposes.” We recommend that the organizations that are subject to PCI DSS use the motivating power of PCI DSS to acquire the needed tools and then proceed to using them for solving all the problems in the whole IT security realm.