CHAPTER 21
Information Assurance Measurements and Metrics
The famous paraphrased statement in Edwards Deming’s book Out of Crisis notes that “what you cannot measure, you cannot improve.” This applies to managing information assurance. The ability to make quantitative judgments and comparisons about information assurance is desirable for continuous improvement. By using the appropriate metrics, an organization will have a basis to determine how and where to allocate its limited resources. Thus, measurements and metrics provide means for an organization to gain a more concrete understanding of the effectiveness of their efforts in securing information.
It is important to establish the difference between measurements and metrics. Measurement refers to a specific single point-in-time snapshot of data, whereas metrics are derived from comparing predetermined baselines against a series of measurements taken over time. Measurement consists of raw data, whereas metrics are interpretations of the data collected through the measurement process.
This chapter discusses the importance of security measurement and metrics. It provides insights into the processes that organizations use to perform the measurements and to manage an information assurance metric program.
Importance of Information Assurance Measurement
Information assurance measurement helps management make more objective decisions about information assurance and identify noncompliance, and it provides valuable indicators of information assurance performance. With the emergence of legislative and regulatory requirements, organizations need to demonstrate to stakeholders that internal controls have been established and that information systems’ assurance posture is adequately managed and monitored.
The use of information assurance measurements makes an organization able to establish the effectiveness of its information assurance controls and processes. Moreover, measurement assists in reporting current and past compliance status to management and forms input for internal audit and management review.
Information Assurance Measurement Process
Foundationally, organizations should base quantifiable information assurance measures on information assurance objectives and policy. Measures should be repeatable and provide a standard process for tracking compliance and performance over time. As illustrated in Figure 21-1, the information assurance measurement process consists of five key steps. The details of the steps are discussed in the next section.
Figure 21-1 Information assurance measurement process
Develop Measurements
The planning stage defines the approach and method for measuring and selecting metrics to support strategy and risk tolerance. Measurement planning includes selecting controls, identifying objectives, creating specifications, establishing data collection, analyzing, and reporting. Planning should identify financial, human, and infrastructure resources required to ensure each task has sufficient resources. You should document planning activities for future reference and to record decisions.
Collect Data
Using well-planned data collection procedures is the difference between successful and unsuccessful information assurance measurement. Data collection should be designed for establishing baselines followed by continuous monitoring. Design the data structures and analytic techniques before data collection begins. The activities ensure the collected data can be used to gain an understanding of an information assurance management system and recommend improvement actions.
U.S. NIST uses the approach in Figure 21-2 when determining what data to collect and from which sources.
Figure 21-2 U.S. NIST measures development process
Analyze and Report
To gain a meaningful understanding of collected data, it has to be analyzed using a predefined measurement method. One of the most common methods of analysis is statistical process control (SPC). Statistical process control uses statistical methods to identify and control processes. SPC is based on the work of Walter A. Shewhart of the Bell Telephone Laboratories in the 1920s and was foundational to Demming’s work in measurement and quality control. Since then it has been used in numerous industries to minimize and eliminate waste and inefficiencies. SPC often uses a tool called control charts to explain the analysis.
SPC focuses on understanding the relationship of outside events and their impacts on a process and is often used for early detection of events and prevention of problems. SPC works best in environments with consistent output of information and a fairly consistent operation. Networks, training, incidents, and system patching are all processes that typically work well with SPC. Ultimately, SPC focuses on continuous improvement and understanding of inputs and outputs.
Integrate Measurement Output
The output of the measurement activities have many purposes including, but not limited to, providing an indicator about control effectiveness, input to risk assessment, and decision making about information assurance. The output is indicative of compliance with requirements as well as benchmarks among business units or organizations in the same or similar industry sectors.
Improve Measurement Process
Review the measurement process and measurements taken periodically to ensure successful implementation and consistent operation. Improvement is required when new controls are introduced or the current process is incapable of capturing data effectively.
The information assurance team can perform a cost-benefit analysis by comparing the usefulness of the results gathered and the cost spent to obtain the results against the projected measurement objectives. Process improvement should drive measures and metrics maturity. Figure 21-3 from U.S. NIST illustrates the maturing of a metrics and measurement program.
Figure 21-3 U.S. NIST program maturity
Importance of Information Assurance Metrics
Information assurance metrics allow an organization to improve its information assurance performance. The development of an information assurance metrics program allows an organization to monitor the status of measured activities, and corrective actions can be applied based on the observations.
Information assurance metrics provides an organization with both operational and financial benefits. From the operational standpoint, using information assurance metrics allows an organization to gauge the adequacy of the information assurance controls in place. With this information, the organization can make financial decisions about investments in additional information assurance protection or termination of non-productive controls.
Information Assurance Metrics Program
Metrics should yield easily acquired quantifiable information. Initially consider processes that are consistent and repeatable, with ease of data collection. While the easiest-to-collect metrics serve as an initial effort, the organization must determine which metrics are critical for its success.
Unstable processes from which information is hard to retrieve defeat the purpose of having a metrics program in place since it results in unnecessary resource allocation and cost.
Figure 21-4 illustrates the key activities for implementing an information assurance metrics program.
Figure 21-4 Implementation of an information assurance metrics program
Data Collection Preparation
During the data collection preparation phase, an organization identifies and selects the processes to be included in the information assurance metrics program. This includes defining how the data will be collected, analyzed, and reported. In addition, you should define and plan the associated roles and responsibilities since they support the tracking and monitoring of the defined information assurance metrics. Often, metrics are chosen because of a “best practice” or industry standard. Organizations must be cautious when adopting these metrics because they may not align with an organization’s key performance indicators (KPIs) or critical success factors.
KPIs are metrics that are critical for an organization’s success. They may include metrics that are not specifically related to information assurance such as the following:
• Return on investment
• Return on assets
• Amount of product in a warehouse
• Age of product in a warehouse
• Lead time of manufacturing
• Cost of goods sold
• Days before an accident
• Number of products returned
• Number of new customers
Initially, these metrics may appear to have little to do with information assurance. A knowledgeable information assurance team will quickly point out the following:
• IT investments will fail if not properly secured, and therefore the return on investment will be greatly decreased.
• If assets are leveraged for an Internet attack (such as a company’s server is hijacked), the return on the asset may actually be negative if the server can’t perform its intended function.
• The cost of goods sold can be decreased by ensuring the IT infrastructure and other overhead functions are running in a secure and mature manner.
• Industrial accidents can be reduced by ensuring any industrial equipment that has a network connection is secured and protected from attackers.
• Products can be returned for a variety of reasons. If an IT product cannot protect the privacy of a customer, there can be backlash.
• New customers can be attracted through trust in an organization’s product and services. The IA team can determine whether the products and services are protecting the customer and building a foundation of trust.
Data Collection and Analysis
The data collection and analysis phase develops an insight to the information assurance controls and corrective actions that need to be taken. Data is collected, consolidated, and compared against the predefined target. This comparison identifies the root cause of poor performance and areas of improvements.
Corrective Action Identification
The corrective actions identification phase consists of the preparation of a plan to mitigate the subpar performance and areas of improvements identified previously. The plan identifies corrective actions, prioritizes them, and recommends actions based on the criticality of the performance issue.
Business Case Development
Finally, develop a business case based on the identified corrective actions. The business case analysis compares and contrasts the cost of the remaining status quo versus the cost of implementing remedies for corrective actions. This helps an organization justify budget and resource allocation.
Corrective Action Applications
Implement the identified corrective actions for the information assurance controls. This establishes the input for the next iteration of the review cycle.
Further Reading
• NIST. What Are Process Control Techniques? www.itl.nist.gov/div898/handbook/pmc/section1/pmc12.htm.
• NIST. Process or Product Monitoring and Control. www.itl.nist.gov/div898/handbook/toolaids/pff/pmc.pdf.
• Panye, SC. A Guide to Security Metrics. SANS Institute, 2006. www.sans.org/reading_room/whitepapers/auditing/55.php.
• Ryan, D., J.C.H. Julie, and C.D. Schou. On Security Education, Training, and Certifications. Information Systems Audit and Control Association, 2004.
• Sademies, S. Process Approach to Information Security Metrics in Finnish Industry and States Institutions. VTT Technical Research Center of Finland, 2004. www.vtt.fi/inf/pdf/publications/2004/p544.pdf.
• Conklin, Wm. Arthur, Introduction to Principles of Computer Security: Security+ and Beyond. McGraw-Hill Education, 2004.
• Schou, Corey D., and D.P. Shoemaker. Information Assurance for the Enterprise: A Roadmap to Information Security. McGraw-Hill Education, 2007.
• Swanson, M., and B. Guttman. Generally Accepted Principles and Practices for Securing Information Technology Systems. NIST, 1996.
• Swanson, M., et al. Security Metrics Guide for Information Technology Systems (Special Publication 800-55). U.S. Government Printing Office, 2003.
Critical Thinking Exercises
1. An organization is struggling. After years of investing in research and development, a competitor appears to have stolen design documents for the organization’s flagship product. The organization’s CISO has been asked to give a presentation to the board regarding the best metrics to monitor to prevent information leakage in the future. What information assurance metrics should the CISO propose?
2. A CIO wants to ensure she is investing properly in information assurance. What metrics should her CISO advise her to monitor?