Key concepts you will need to understand:
✓ Knowledge of crisis management and business impact analysis techniques
✓ Knowledge of disaster recovery and business continuity planning and processes
✓ Knowledge of backup and storage methods and practices
✓ Knowledge of disaster recovery and business continuity testing approaches and methods
✓ Knowledge of insurance in relation to business continuity and disaster recovery
✓ Knowledge of human resource issues (such as evacuation planning and response teams)
Techniques you will need to master:
✓ Evaluate the adequacy of backup and recovery provisions to ensure the resumption of normal information processing in the event of a short-term disruption or the need to rerun or restart a process
✓ Evaluate the organization’s capability to continue to provide information system–processing capabilities in the event that the primary information-processing facilities are not available
✓ Evaluate the organization’s capability to ensure business continuity in the event of a business disruption
Disaster recovery for systems typically focuses on making alternative processes and resources available for transaction processing. A disaster recovery plan (DRP) should reduce the length of recovery time necessary and also the costs associated with recovery. Proper planning will mitigate the risk and impact of a major business interruption. Although DRP results in an increase of pre- and post-incident operational costs, the extra costs are more than offset by reduced recovery and business impact costs. A disaster can be classified as a disruption that causes critical information resources to be inoperative for a period of time, adversely affecting business operations. Business continuity plans (BCP) are the result of a process of plan creation to ensure that critical business functions can withstand a variety of emergencies. Disaster-recovery plans deal with the immediate restoration of the organization’s business systems while the business continuity plan also deals with the long-term issues before, during, and after the disaster. The BCP should include getting employees to the appropriate facilities; communicating with the public, partners, and customers; and making the transition from emergency recovery back to normal operations. The DRP is a part of the BCP and is the responsibility of senior management.
A disaster can be caused by naturally occurring events such as floods, tornadoes, fire, or earthquakes, but it can include anything that causes disruption to information processing. Other types of disasters include loss of electrical power or telecommunications, or direct or indirect attacks on the organization’s systems or facilities (such as a terrorist attack or hacking). These are the attributes of a disaster:
Unplanned and unanticipated
Impacts critical business functions
Has the capacity for significant loss
According to the United Nation’s International Decade for Natural Disasters Reduction, natural disasters kill one million people around the world each decade and leave millions more homeless each year. In addition, economic damages from natural disasters have tripled in the past 30 years, rising from $40 billion in the 1960s to $120 billion in the 1980s. In the past year, more than a dozen worldwide disasters have caused billion-dollar losses. Table 5.1 provides a snapshot of the costs resulting from natural disasters from 1983 to 1994.
Table 5.1. Costs of Natural Disasters from 1983 to 1994
Source: World Health Organization | |
|---|---|
Hurricane Alicia (USA, 1983) | $1.65 billion |
Winter storm Herta (Europe, 1990) | $1.90 billion |
Forest fire (USA, 1991) | $2 billion |
Winter storm Wiebke (Europe, 1990) | $2.25 billion |
Hurricane Iniki (Hawaii, 1992) | $3.00 billion |
Winter storm Vivian (Europe, 1990) | $3.25 billion |
Winter gale (Western Europe, 1987) | $3.70 billion |
Blizzard (USA, 1993) | $5.00 billion |
Typhoon Mireille (Japan, 1991) | $6.00 billion |
Winter storm Daria (Europe, 1990) | $6.80 billion |
Hurricane Hugo (USA, Caribbean, 1989) | $9.00 billion |
Floods (USA, 1993) | $12.00 billion |
Northridge Earthquake (USA, 1994) | $30.00 billion |
Hurricane Andrew (USA, 1991) | $30.00 billion |
During the initiation of the business continuity planning process, the BCP team should prepare for a meeting with senior management to define the project goals and objectives, present the project schedule, and review the proposed interview schedule (resources required). In preparation for this meeting, the BCP team should do the following:
Review the organizational structure to determine what resources will be assigned to the team
Review existing disaster-planning policies, strategies, and procedures
Review existing continuity plans
Research any events that have occurred previously (severe weather, fires, equipment or facility failures, and so on) and that had or could have a negative effect on the organization
Create a draft project schedule and associated documents (timing, resources, interview questionnaires, roles and responsibilities, and so on)
Per ISACA, the business continuity planning process can be divided into the following phases:
Analyze the business impact
Develop business-recovery strategies
Develop a detailed plan
Implement the plan
Test and maintain the plan
The development of an effective business-continuity plan will take all threats (disasters) into account during development. Some of these threats might affect systems only for minutes or hours, but the plan should include recovery from these events as well. The recovery might be simply restoring data from backups or moving personnel and equipment to a new facility to continue business operations.
A business impact analysis (BIA) is used to identify threats that can impact continuity of operations. These threats might be natural or man-made and should encompass everything from a telecommunications outage to a fire or hurricane. The results of the BIA should provide a clear picture of the continuity impact in terms of the impact to human and financial resources, as well as the reputation of the organization. To assess the risks associated with continuity, the BIA team should have a clear understanding of the organization, key business processes, and IT resources that support those processes. The BIA team should work with senior management, IT personnel, and end users to identify all resources used during normal operations. These resources might include both automated and manual processes. Although BCP and DRP are often implemented and tested by middle management and end users, the ultimate responsibility and accountability for the plans remains with executive management, such as the board of directors. The following steps can be used for the framework of business impact assessment:
Gather business impact analysis data
Questionnaires or interviews
Review the BIA results
Check for completeness and consistency
Follow up with interviews for areas of ambiguity or missing information
Establish the recovery time for operations, processes, and systems
Define recovery alternatives and costs
The BIA will help the organization understand the degree of loss associated with the business functions and associated systems. This covers financial loss as well as loss of customer confidence and damage to the organization’s reputation. The BIA questionnaire and interviews should gather the following information from the business units:
Financial impacts resulting from the incapability to operate for prolonged periods of time
Operational impacts within each business unit
Expenses associated with continuing operations after a disruption
Current policies and procedures to resume operations in the event of a disruption
Technical requirements for recovery
The BIA should include both quantitative and qualitative questions. Quantitative questions generally describe the economic or financial impacts of a potential disruption. These types of disruptions are measured in monetary terms, including both loss of income and expenses incurred during and after recovery. Quantitative impacts might include loss of revenue or sales, interest paid on loans, penalties for late payments to vendors, fines or penalties associated with contractual obligations, unavailability of operating funds, delayed or canceled orders, and so on. Expenses might include use of third-party services, emergency purchases related to recovery, rental or lease equipment, and relocation of employees. Qualitative impacts are impacts that cannot be quantified in monetary terms. These types of impacts are generally associated with the business impact of a disaster and include damage to reputation and loss of confidence in customer services or products. Although DRP results in an increase of pre- and post-incident operational costs, the extra costs are more than offset by reduced recovery and business impact costs.
A couple approaches can be taken to a BIA: The team might develop questionnaires for senior management and end users, or might gather information during an interview process. The important part of the process is to identify, sequence, and prioritize mission-critical processes. During the information-gathering phase of the BIA, the team will generally get information from individual business units. In addition to the information gathered from the business units, the team should identify the IT resources required for each process and the current disaster-recovery procedures. When the questionnaires are complete, the BCP team should conduct interviews to clarify information contained in the questionnaires, to ensure that the organization has identified time-sensitive business operations and services, financial risks, correct time frames for the resumption of operations, and estimates of the resources required for successful recovery. A sample of a BIA questionnaire is shown in Figure 5.1.
Table 5.1. BIA questionnaire
Organization: | Date Complete: |
Business Unit: | BIA POC: |
| |
Identification of Business Unit Function - Description of the function being performed. | |
Function Dependencies - Description of the dependencies of the function. | |
Business Records – What business records are needed, and are they automated or manual? If required, are they backed up? How often? | |
| |
Financial Impacts – What and when would the financial impact be to the business if the function was not performed? | |
Operational Impacts – What and when would the operational impact be to the business if the function was not performed? | |
Business Disruption – Has the business unit experienced a disruption in the past? What type of disruption? How was it handled (recovery, operations, etc)? | |
| |
Recovery - What type of resources are needed to support the function, how many are needed, and how soon are they needed after a disruption (personnel, office space, telephones, etc)? | |
Identify System Resources - What technology resources are required to support the function (include quantity and type)? | |
Hardware/Software | |
When the BIA questionnaires and the interviews are complete, the BIA team should begin to document the results in the form of a BIA recommendation report. This report should allow for the prioritization of recovery among the business functions, and also give the team an overall view of potential recovery scenarios within the organization. This overall view might highlight gaps where additional information is required. When the initial draft is complete, the BIA team should develop a summary sheet to send back to the interviewees for confirmation. This allows the interviewees to review the information and add information that might have come up since the initial questionnaire or interview. The BIA is an important step in business continuity planning because all future decisions are based on the information gathered during the BIA. It is important to ensure that the information is as accurate as possible and that individual business units and end users are closely tied to the development of the business continuity plan.
During the creation of the recommendation report, the BIA team must define time-critical business functions and processes and their interdependencies among the business units. The development of recovery scenarios depends on the clear definition of time-critical processes and the financial and operational impacts gathered during the BIA. Before the development of a BCP/DRP, the BIA team should develop a recommendation or findings report for senior management. The purpose of this report is to provide senior management with a draft priority list of the business unit service and support recovery, as well as the financial and operational impacts that drive the prioritization. This step will give senior management the opportunity to approve the recovery priorities and prepare them for the next phase, in which they will review the recovery solutions and associated costs.
The objective of a BCP is to ensure that the organization can continue operations and keep the costs associated with both downtime and recovery to a minimum. In reviewing the information gathered during the BIA, the team should determine what the critical information resources are related to the organization’s critical business processes. This relationship is important because the disruption of an information resource is not a disaster unless that resource is critical to a business process. Per ISACA, each resource should be assessed to determine criticality. Indications of criticality might include these:
The process supports lives or people’s health and safety.
Disruption of the process would cause a loss of income to the organization or exceptional costs that are unacceptable.
The process must meet legal or statutory requirements.
An important factor is the time period in which critical information resources must resume processing before significant or unacceptable losses are suffered. These time periods will depend on the type of business. As an example, the technology resources (hardware, software, network, and so on) that are used in completing stock transactions would probably be deemed critical, and the disruption or delay in resumption of any component of these services would result in large financial losses for that organization. In contrast, a smaller organization, such as a nonprofit organization, might be able to go without technology resources for hours or a few days without significant impact to the organization.
In making this determination, the BIA team should consider two cost factors. The first is the cost associated with downtime. This cost is defined in terms of hours per days, and the cost usually increases quickly over time to a certain point at which it stops growing. The stop in growth reflects the point in time when the business can no longer function. The costs associated with downtime vary based on the organization but might include a drop in order transactions, the cost of idle resources, the cost associated with the incapability to invoice customers or collect billing information, and qualitative costs associated with damage to reputation, goodwill, or the loss of market share. The second cost factor is the cost associated with recovery or resumption of services by implementing the business continuity plan. These costs include the cost of the development and maintenance of the continuity plan, off-site premises, insurance, and resources associated with recovery and resumption. As stated earlier, an optimal BCP and associated strategies should be based on the point in time when both cost factors are at a minimum. As an example of balancing these costs, the business might be capable of sustaining a longer recovery time, which will generally be less expensive but will incur more downtime costs than a shorter recovery. The combination of these costs should be taken into consideration when developing the recovery strategies.
The BIA is used to help business units understand the impact of a disruptive event and should include the execution of a vulnerability assessment for critical business processes to identify natural, man-made, and technical threats. The implementation of the BIA requires a high level of support from senior management and requires extensive involvement from IT and end-user personnel. The information collected during the BIA is used to develop the actual business continuity plan, which includes plan implementation, testing, and maintenance.
The next step in developing the business continuity plan is to identify recovery strategies and select the strategy or strategies that best meet the organization’s needs. It is important to remember that the strategy should include the technologies required for recovery and that the policies and procedures should include specific sequencing. The sequence in which systems are recovered is important for ensuring that the organization can function effectively following a disaster. As an example, the organization might need access to the accounting systems and associated accounting functions to facilitate the purchase of equipment associated with a recovery. If the accounting personnel and systems are not brought online first, this could delay the recovery process. Using the results of the BIA, the BCP team should identify both manual and automated processes that are required for the organization to resume business operations. These processes might include notifying personnel and moving them to processing facilities; notifying partners, customers, and shareholders of a disaster; and bringing hardware, software, and data online for use in processing.
Per ISACA, the classification matrix shown in Table 5.2 can be used to classify the criticality of systems to be recovered. This matrix will help the BCP team identify the best recovery strategies and alternative recovery strategies to be presented to senior management. The selection of the recovery strategy is based on the following:
The criticality of the business process and the applications supporting the process
The cost of the downtime and recovery
Time required to recovery
Security
Table 5.2. System Classification
A variety of strategies exist for the recovery of critical business processes and their associated systems. The best strategy is one that takes into account the cost of downtime and recovery, the criticality of the system, and the likelihood of occurrence determined during the BIA. In addition to actual recovery procedures, the organization should implement different levels of redundancy so that a relatively small event does not escalate to a full-blown disaster. An example of this type of control is to use redundant routing or fully meshed wide area networks. This redundancy would ensure that network communication will continue if portions of the wide area network are lost. This type of redundancy acts to either remove the threat altogether or minimize the likelihood or the effect of occurrence. These types of controls should be evaluated when developing the business-recovery strategies.
The recovery solution might include the use of different types of physical processing facilities and should include agreements and the costs associated with the facility both before and during use.
A hot site is a facility that is basically a mirror image of the organization’s current processing facility. It can be ready for use within a short period of time and contains the equipment, network, operating systems, and applications that are compatible with the primary facility being backed up. When hot sites are used, the staff, data files, and documentation are the only additional items needed in the facility. A hot site is generally the highest cost among recovery options, but it can be justified when critical applications and data need to resume operations in a short period of time. The costs associated include subscription costs, monthly fees, testing costs, activation costs, and hourly or daily charges (when activated). The use of a hot site generally includes connectivity over public networks (WAN or Internet) to enable regular backups and periodic testing to ensure that the hardware and software are compatible.
As with any recovery plan, the hot site should be part of the testing and maintenance procedures. The organization will incur costs associated with a live recovery, which requires the organization’s personnel to work onsite at the hot site facility to test the recovery of applications and data. Generally, hot sites are to be used for a relatively short recovery time; they would be used only for a period of a week to several weeks while the primary facility is repaired. The physical facility should incorporate the same level of security as the primary facility and should not be easily identifiable externally (with signs or company logos, for example). This type of external identification creates an additional vulnerability for sabotage. In addition, this facility should not be subject to the same natural disaster that could affect the originating site and, thus, should not be located in proximity to the original site.
Warm sites are sites that contain only a portion of the equipment and applications required for recovery. In a warm site recovery, it is assumed that computer equipment and operating software can be procured quickly in the event of a disaster. The warm site might contain some computing equipment that is generally of a lower capacity than the equipment at the primary facility. The contracting and use of a warm site are generally lower cost than a hot site but take longer to get critical business functions back online. Because of the requirement of ordering, receiving, and installing equipment and operating systems, a warm site might be operational in days or weeks, as opposed to hours with a hot site. The costs associated with a warm site are similar to but lower than those of a hot site and include subscription costs, monthly fees, testing costs, activation costs, and hourly or daily charges (when activated).
A cold site can be considered a basic recovery site, in that it has the required space for equipment and environmental controls (air conditioning, heating, power, and so on) but does not contain any equipment of connectivity. A cold site is ready to receive the equipment necessary for a recovery but will take several weeks to activate. Of the three major types of off-site processing facilities (hot, warm, and cold), a cold site is characterized by at least providing for electricity and HVAC. A warm site improves upon this by providing for redundant equipment and software that can be made operational within a short time.
Duplicate processing facilities are similar to hot site facilities, with the exception that they are completely dedicated, self-developed recovery facilities. An example of duplicate processing facilities is large organizations that have multiple geographic locations. The organization might have a primary site in Washington, D.C., and might designate a duplicate site at one of its own facilities in Utah. The duplicate facility would have the same equipment, operating systems, and applications and might have regularly synchronized data. In this example, the facility can be activated in a relatively short period of time and does not require the organization to notify a third party for activation. Per ISACA, several principles must be in place to ensure the viability of this approach:
The site chosen should not be subject to the same natural disaster(s) as the original (primary) site.
There must be a coordination of hardware and software strategies. A reasonable degree of compatibility must exist to serve as a basis for backup.
Resource availability must be ensured. The workloads of the sites must be monitored to ensure that availability for emergency backup use will not be impaired.
There must be agreement on the priority of adding applications (workloads) until the recovery resources are fully utilized.
Regular testing is necessary. Even though duplicate sites are under common ownership, and even if the sites are under the same management, testing of the backup operation is necessary.
Reciprocal agreements are arrangements between two or more organizations with similar equipment and applications. In this type of agreement, the organizations agree to provide computer time (and sometimes facility space) to one another in the event of an emergency. These types of agreements are generally low cost and can be used between organizations that have unique hardware or software that cannot be maintained at a hot or warm site. The disadvantage of reciprocal agreements is that they are not enforceable, hardware and software changes are generally not communicated over time (requiring significant reconfiguration in the event of an emergency), and the sites generally do not employ capacity planning, which may render them useless in the event of an emergency. ISACA recommends that organizations considering a reciprocal agreement ensure the terms of the agreement by answering the following questions:
How much time will be available at the host computer site?
What facilities and equipment will be available?
Will staff assistance be provided?
How quickly can access be gained to the host recovery facility?
How long can the emergency operation continue?
How frequently can the system(s) be tested for compatibility?
How will the confidentiality of data be maintained?
What type of security will be afforded for information systems operations and data?
How much advance notice is required for using the facility?
Are there certain times of the year or month when the partner’s facilities are not available?
In reviewing the recovery options, the BCP team should review both the agreements and the facilities to be used in recovery to ensure that they will meet the demands of the organization. The facility should have the capacity (space, network, and infrastructure) to support a recovery and should not be oversubscribed. If a facility is oversubscribed and multiple companies declare a disaster at or near the same time, the facility would not be capable of supporting recovery. The vendor that owns the facility should be able to attest to the reliability of the site to include UPS, number of subscribers, diverse network connectivity, and guarantees of space and availability.
The organization must define procedures and put in place agreements to ensure that needed hardware and software will be available. This might include the use of emergency credit lines or credit cards with banks, agreements with hardware and software vendors, and agreements for backup data. A majority of hardware vendors provide high-response services that guarantee hardware and software availability times. These agreements must be in place before the declaration of an emergency. If the organization maintains off-site backup media, there should be an agreement in place for the procurement and shipping of media to the recovery facility.
The BCP team should develop a detailed plan for recovery. This plan should include roles and responsibilities as well as specific procedures associated with the recovery. The following factors should be considered when developing the detailed plan:
Predisaster readiness: Contracts, maintenance and testing, policies, and procedures
Evacuation procedures: Personnel, required company information
Disaster declaration: What defines a disaster? Who is responsible for declaring?
Identification of critical business processes and key personnel (business and IT)
Plan responsibilities: Plan objectives
Roles and responsibilities: Who is responsible for what?
Contract information: Who maintains it, and where is it?
Procedures for recovery: Step-by-step procedures with defined responsibilities
Resource identification: Hardware, software, and personnel required for recovery
The BCP should be written in clear, simple language and should be understandable to all in the organization. It is important to remember that the plan will be implemented under the worst of circumstances, personnel who are assigned duties may not be available, and those who are available could be under significant emotional stress. When the plan is complete, a copy should be maintained off-site and should be easily accessible.
When the primary components of the plan are in place, it is time to organize the plan. The plan should be organized to address response, resumption, recovery, and restoration. The resources required for a successful recovery include the following:
People—. Team members, vendors, partners, customers, clients, shareholders, employees, and services
Places—. Alternative recovery sites, processing locations, off-site storage facilities, vaults, and so on
Things—. Supplies, equipment (computing, office, voice and data communications), and vital records (data, software, documentation, forms, contracts)
The organization of the plan should be prepared to define step-by-step procedures that will take place when a disaster is declared and notification of the necessary personnel who are responsible for the timely resumption of critical business processes and systems. During the organization, the BCP team should incorporate existing policies, procedures, and recovery plans. In addition, the team should define specific training for both key personnel (BCP teams) and employees.
The business continuity plan should be created to minimize the effect of disruptions. The process associated with the development of the plan should include the following steps:
Perform a business impact analysis to determine the effect of disruptions on critical business processes
Identify, prioritize, and sequence resources (systems and personnel) required to support critical business processes in the event of a disruption
Identify recovery strategies that meet the needs of the organization in resumption of critical business functions until permanent facilities are available
Develop the detailed disaster-recovery plan for the IT systems and data that support the critical business functions
Test both the business continuity and disaster recovery plans
Maintain the plan and ensure that changes in business process, critical business functions, and systems assets, such as replacement of hardware, are immediately recorded within the business continuity plan
As an IS auditor, you should review the plan to ensure that it will allow the organization to resume its critical business functions in the event of a disaster. ISACA states the IS Auditors tasks include the following:
Evaluating the business continuity plans to determine their adequacy and currency, by reviewing the plans and comparing them to appropriate standards or government regulations
Verifying that the business continuity plans are effective, by reviewing the results from previous tests performed by both IT and end-user personnel
Evaluating off-site storage to ensure its adequacy, by inspecting the facility and reviewing its contents, security, and environmental controls
Evaluating the ability of IT and user personnel to respond effectively in emergency situations, by reviewing emergency procedures, employee training, and results of their tests and drills
The backup of both software and data varies among organizations, and both the methods and technology used in backing up data and software will affect recovery time. The organization’s critical data should be stored both onsite, for quick recovery in nondisaster situations, and off-site, in case of a disaster. The Storage Networking Industry Association defines a backup as follows:
A collection of data stored on (usually removable) nonvolatile storage media for purposes of recovery in case the original copy of data is lost or becomes inaccessible. Also called a backup copy. To be useful for recovery, a backup must be made by copying the source data image when it is in a consistent state... or contains elements and information enabling a consistent state to be recovered.
Organizations continue to rely on the availability of computer services and corporate data. The IT department is responsible for ensuring that systems and data are available, and that the organization is capable of recovering from disasters, to enable continuity of operations. There are a variety of threats to systems and data, ranging from the accidental deletion of corporate data to a disaster that affects the physical facilities and the systems contained within that facility. The evolution of the corporate computing environment has led to tighter integration of systems and applications. In this environment, the database, file, web, communications, and messaging servers are components of a larger system. A failure of any component affects the system as a whole. Most organizations have implemented a centralized backup scheme that incorporates enterprise backup software, tape libraries, and specific storage requirements. To ensure minimum downtime, it is important to understand the different types of backups and their effect on recovery time.
Three backup methods are used:
Full backup—. In a full backup, all the files (in some cases, applications) are backed up by copying them to a tape or other storage medium. This type of backup is the easiest backup to perform but requires the most time and space on the backup media.
Differential backup—. A differential backup is a procedure that backs up only the files that have been changed or added since the last full backup. This type of backup reduces the time and media required.
Incremental backup—. An incremental backup is a procedure that backs up only the files that have been added or changed since the last backup (whether full or differential).
The method of backup depends on factors that include the cost of media, the speed of restoration, and the time allocated for backups. For instance, the organization might choose to perform a single full weekly backup combined with daily incremental backups. This method decreases the time and media required for the daily backups but increases restoration time. This type of restoration requires more steps and, therefore, more time because the administrator will have to restore the full backup first and then apply the incremental backups sequentially until all the data is restored.
A variety of vendors provide centralized enterprise backup software, and their products generally work off the same basic premise—that is, to back up systems over the network to a server that has some sort of storage device attached. Generally, a central server controls the enterprise backup environment. The backup software incorporates backup schedules, indexes, backup groups, and communication with the client software. In addition, the central server logs its activities to include communication, backup start and end times and dates, and any errors incurred during the backup. To effect communication with clients, a backup agent (client software) is installed on all systems that will be backed up through the central server. The client software listens for connections from the central server and assists in the transfer of data from the client to the central server.
Tape backup media is a magnetic medium and, as such, is susceptible to damage from both the environment in which it is stored (temperature, humidity, and so on) and physical damage to the tape through excessive use. For this reason, administrators use backup schemes that allow tapes to be regularly rotated and eventually retired from backup service. One popular scheme is the grandfather, father, and son scheme (GFS), in which the central server writes to a single tape or tape set per backup. When using the GFS scheme, the backup sets are daily (son), weekly (father), and monthly (grandfather). Daily backups come first. The four backup tapes are usually labeled (Mon–Thur) and used on their corresponding day. The tape rotation is based on how long the organization wants to maintain file history. If a file history for one week is required, tapes are overwritten each week; if history is required for three weeks, each tape is overwritten every three weeks (requiring 12 tapes). The five (some months have five weeks) father tapes are used for full weekly backups (Friday tapes). If one month of history is being kept, tapes are overwritten monthly. The three grandfather tapes are used as full monthly backups and are typically overwritten quarterly or yearly.
Based on its retention/rotation, the tape is then retained for a period of time; when the tape has reached its expiration date, it can be put back into the rotation and used again. One of the disadvantages of this scheme is that sometimes the full capacity of the media is not used. As an example, if the administrator is using an 80GB tape that backs up 25GB of data, the tape will be rotated out, and when it expires, it will be rewritten from the beginning (with 25GB), leaving the remaining 55GB unused. All tapes within the backup scheme will be saved based on the retention period assigned to them. Creating a retention schedule ensures that an organization maintains historical records for an appropriate period of time, in compliance with business requirements and any regulations pertaining to business operations. This retention schedule also ensures that unnecessary records are disposed of in a controlled manner. A retention schedule should include all the types of records, period of retention, description of the records, disposition (destroy, transfer, and so on), and retention requirement.
Two types of tape storage are used:
Onsite storage—. One copy of the backup tapes should be stored onsite to effect quick recovery of critical files. Another copy should be moved to an off-site location as redundant storage. Tapes should be stored in environmentally controlled facilities that incorporate physical access controls that are commensurate with the requirements of the data being stored. Onsite tapes should be stored in a secure fireproof vault, and all access to tapes should be logged.
Off-site storage—. The organization could contract with a reputable records storage company for off-site tape storage, or could maintain the facility themselves. The physical and environmental controls for the off-site facility should be equal to those of the organization. The contract should stipulate who from the organization will have the authority to provide and pick up tapes, as well as the time frame in which tapes can be delivered in the event of a disaster.
In addition to tape backup options, organizations can employ storage area networks (SAN) or electronic vaulting options. A SAN is a special-purpose network in which different types of data storage are associated with servers and users. A SAN can either interconnect attached storage on servers into a storage array or connect the servers and users to a storage device that contains disk arrays. The SAN can be implemented locally or can use disk arrays at a redundant facility. The enterprise backup software either can back up the entire array to a separate storage medium or, in the case of an off-site SAN, can instruct the SAN itself to create a snapshot of the local volumes and then move the snapshot to the off-site SAN.
If the organization cannot implement an off-site SAN, it might opt for an electronic vaulting option. With this option, the organization contracts with a vaulting provider that provides disk arrays for the backup and storage of the organization’s applications and data. Generally, the organization installs an agent on all the servers and workstations that require a backup and identifies the files to be included in the backup. The agent then performs full and incremental backups, and moves that data via a broadband connection to the electronic vault. Organizations that have a significant amount of data or high levels of change might incur issues in moving large amounts of data across a broadband connection.
As with all IT procedures, proper security should be implemented to enforce segregation of duties and ensure the integrity of the backup media and data. The backup administration should be responsible for backup scheduling and adding machines and drives to the backup schedule. A tape operator should be responsible for adding and removing tapes from the various devices and tape libraries in a data center, but should not be allowed access to change client definitions or backup schedules. A systems operator should be responsible for checking backup status and ensuring that the central server’s OS is up-to-date and operating correctly.
The objective of having backups is to ensure recovery in the event of a failure or disaster. The organization should perform regular disaster-recovery testing to ensure that data can be restored within the time frame required in the BCP. The organization should utilize off-site storage facilities to maintain redundancy of current and critical information within backup files. The off-site data backup and storage should be geographically separate, to mitigate the risk of a widespread physical disaster such as a hurricane or earthquake.
As a part of regular testing and maintenance, organizations can opt to perform either full or partial testing of recovery and continuity plans, though most organizations do not perform full-scale tests because of resource constraints. To continue to improve recovery and continuity plans, organizations can perform a paper, walk-through, or preparedness test. Tests should be scheduled during a time that causes minimal disruption to the normal operations of the organization. It is important that all key team members participate in testing and that the test process addresses all critical areas of the plan. The testing methods employed by the organization will vary from simple to complex, and each method has its own objectives and benefits. The following sections give examples of testing methods.
A paper test is the least complex test that can be performed. This test helps ensure that the plan is complete and that all team members are familiar with their responsibilities within the plan. With this type of test, the BCP/DRP plan documents are simply distributed to appropriate managers and BCP/DRP team members for review, markup, and comment.
A walk-through test is an extension of the paper testing, in that the appropriate managers and BCP/DRP team members actually meet to discuss and walk through procedures of the plan, individual training needs, and clarification of critical plan elements.
A preparedness test is a localized version of the full test in which the team members and participants simulate an actual outage or disaster and simulate performing the steps necessary to effect recovery and continuity. This test can be performed against specific areas of the plan instead of the entire plan. This test validates response capability, demonstrates skills and training, and practices decision-making capabilities. Only the preparedness test actually takes the primary resources offline to test the capabilities of the backup resources and processing.
A full operational test is the most comprehensive test and includes all team members and participants in the plan. The BCP team and participants should have multiple paper and preparedness tests completed before performing a full operational test. This test involves the mobilization of personnel, and disrupts and restores operations just as an outage or disaster would. This test extends the preparedness test by including actual notification, mobilization of resources, processing of data, and utilization of backup media for restoration.
Per ISACA, the test should strive to accomplish the following tasks:
Verify the completeness and precision of the business continuity plan
Evaluate the performance of the personnel involved in the exercise
Appraise the training and awareness of the nonbusiness continuity members
Evaluate the coordination among the business continuity team and external vendors and suppliers
Measure the capability and capacity of the backup site to perform prescribed processing
Assess the vital records retrieval capability
Evaluate the state and quantity of equipment and supplies that have been related to the recovery site
Measure the overall performance of operational and information systems–processing activities related to maintaining the business entity
During the test, detailed documentation and observations should be maintained. This documentation should include any problems incurred and suggested solutions. This documentation should be used during analysis of the test, with the success of the plan measured against plan objectives. During this analysis, team members and management should be able to evaluate against specific or general measurements associated with the plan. Per ISACA, these measurements might include the following:
Time—. The elapsed time for completion of prescribed tasks, delivery of equipment, assembly of personnel, and arrival at a predetermined site.
Amount—. Amount of work performed at the backup site by clerical personnel and information systems processing operations.
Count—. The number of vital records successfully carried to the backup site versus the required number, and the number of supplies and equipment requested versus those actually received. Also, the number of critical systems successfully recovered can be measured with the number of transactions processed.
Accuracy—. Accuracy of the data entry at the recovery site versus normal accuracy. Also, the accuracy of actual processing cycles can be determined by comparing output results with those for the same period processed under normal conditions.
It is important for organizations to remember that a BCP plan is a living document and will change according to the needs of the organization. The testing, maintenance, and analysis will provide the organization with a BCP plan that is viable in the event of a disaster. The plan should include a regular review and testing schedule to allow for changes in business strategy, the introduction of new applications, vendor or contract changes, and the disposition of applications or systems. The organization should appoint a business continuity coordinator to ensure that periodic testing and maintenance of the plan are implemented. The coordinator should also ensure that team members and participants receive regular training associated with their duties in the BCP and maintain records and results of testing.
The organization should implement an independent party (internal or external IS auditor) to review the adequacy of the business continuity process, to ensure that the board and management expectations are met. The independent review should include assessing the identification of critical business processes, team and individual skill sets, testing scenarios and schedules, and the communication of test results and recommendations. The IS auditor should directly observe tests and training, and report on the effectiveness of the BCP.
In reviewing the organization’s business continuity planning process, the IS auditor should look for evidence of a structured process in developing the business continuity plan. The planning process should include identifying and prioritizing resources and systems that are required to maintain continuity of critical business processes and strategies for recovery. Senior management is responsible for ensuring that the plan reduces the organization’s risk associated with an unexpected disruption of critical business functions. During the audit, you should review test plans as well as the results of previous tests to ensure the adequacy of the BCP. The BCP should define key personnel and their tasks. Key personnel should have a clear understanding of their tasks and should have detailed documentation on how to perform those tasks.
As an IS auditor, you should review the BCP for adequacy and currency by reviewing the plans and possibly participating in plan testing or reviewing the results of previous tests. In addition, the IS Auditor should review procedures associated with backups to ensure that systems required for critical business processes are included along with storage (onsite and off-site), rotation, and retention procedures. The IS Auditor should also review individual team members to ensure that their skill sets are adequate to perform their duties as described in the plan. Team members should have training specific to these duties, and personnel within the organization should be trained on their roles and responsibilities in the event of a disaster.
Per ISACA, the audit procedures for BCP review include the following:
Obtaining a current copy of the business continuity plan or manual.
Sampling the distributed copies of the manual and verifying that they are current.
Evaluating the effectiveness of the document procedures for the initiation of the BCP.
Reviewing the identification, priorities, and planned support of critical applications, including PC-based or end user–developed systems.
Determining whether all applications have been reviewed for their level of tolerance in the event of a disaster.
Determining whether all critical applications (including PC applications) have been identified.
Determining whether the hot site (if required) has the correct versions of all system software. Also, verifying that all the software is compatible; otherwise, the system will not be capable of processing production data during the disaster recovery.
Reviewing the list of BCP personnel, emergency alternate site contacts, vendor contacts, and so on for appropriateness and completeness.
Calling a sample of the people indicated and verifying that their phone numbers and addresses are correct, as indicated, and that they possess a current copy of the BCP.
Interviewing them for an understanding of their assigned responsibilities in a disaster situation.
Evaluating the procedures for documenting tests.
Evaluating the procedure for updating the manual. Are updates applied and distributed in a timely manner? Are specific responsibilities for maintenance of the manual documented?
The currency and viability of the plan are important, and the IS auditor should ensure that the business continuity coordinator performs regular tests of the plan and updates the plan to mitigate weaknesses discovered during testing. In addition, you will need to ensure that the tests are thorough and performed often enough to incorporate changes in strategy and critical business functions. All contracts associated with the business continuity plan should be included in a regular review, to ensure that response times, capacity, and security procedures are in accordance with the business continuity plan. The purpose of business continuity planning and disaster-recovery planning is to mitigate, or reduce, the risk and impact of a business interruption or disaster.
Business disruptions, as opposed to disasters, can be caused by a variety of internal and external factors, including these:
Equipment failure (processors, hard drives, memory, and so on)
Service failures (telecommunications outages, power outages, external application failure, and so on)
Application or data corruption
In addition to the disaster-recovery plan, the IT department should have policies and procedures for backup, storage of backup media (onsite and off-site), defined roles and responsibilities, and recovery. The IS auditor should review the following to ensure that the organization can recover data and applications in the event of a short-term disruption:
Backup procedures—. The procedures identify the backup scheme and define responsibilities for implementing backups. The procedures should identify how often (weekly or daily) backups are performed, as well as the type of backup (full, differential, or incremental). In addition, the plan should include a retention and rotation schedule to ensure that critical data is in compliance with internal and external guidelines and that tapes are rotated to reduce the chance of error from overuse.
Onsite storage—. All storage media should be stored in environmentally controlled facilities and should be secured in a fire rated safe. Procedures should exist for the inventory of all onsite storage media as well as physical access controls and logging of media check-in and check-out. All storage media should have a record of information regarding the contents, version, and location of data.
Off-site storage—. The off-site storage facility should have environmental and security controls that equal those of the onsite storage facility. The contract with the off-site facility should contain the points of contact within the organization that have the authority to check storage media in and out of the facility, as well as clearly defined response times for the delivery of storage media in the event of a disaster. An inventory of all storage media at the off-site facility should be maintained and should include the dataset name, the volume serial number, the date created, the accounting period, and the off-site storage bin number.
In addition, the plan should include procedures for the restoration of hardware, operating systems, applications, and data. The IS auditor should review all contracts associated with hardware, software, or services, to ensure that the service-level agreements are in accordance with recovery times and that specific points of contact for both the third party and the organization are accurate and up-to-date. All contracts associated with hardware replacement should identify response times to get replacement hardware onsite, support levels, and escalation procedures. The IS Auditor should review previous tests to ensure that the restoration of applications and data meets time requirements of the critical business functions. All documents associated with recovery or restoration should be stored off-site and kept up-to-date in the event of a facility failure.
Although some business continuity plans focus on the procedures regarding major disasters, the recovery of minor disruptions should not be overlooked during planning. The lack of proper backup and restoration procedures associated with a minor disruption can allow the disruption to escalate to a major disruption that may affect the organization’s critical business processes.
The off-site facility should have the same level of access control and security as the originating site. This should include physical access controls such as locked doors and human surveillance. The off-site facility should not be easily identified from the outside (with signs, for example) and should not be subject to the same natural disaster that could affect the originating site. The organization should have procedures associated with the notification and transportation of personnel and the procurement of the necessary hardware, software, and data. The off-site facility should have the same environmental monitoring and controls of the originating site. Per ISACA, the following questions can be considered in reviewing the off-site facility:
Does the plan adequately address the movement to the recovery site?
Does the plan include the items necessary for the reconstruction of the information-processing facility, such as blueprints, hardware inventory, and wiring diagrams?
Does the plan identify rendezvous points for the disaster-management committee or emergency-management team to meet and decide whether business continuity should be initiated?
Does the plan address relocation into a new information-processing facility in the event that the original center cannot be restored?
Is there adequate documentation to perform a recovery?
Does the alternative site contract meet the recovery needs of the organization?
Is the organization’s agreement clear with rules that apply to sites shared with other subscribers? The following rules apply to these sites:
Ensure that insurance coverage ties in with and covers all (or most) expenses of the disaster
Ensure that tests can be performed at the site at regular intervals
Review and evaluate communications requirements for the site
Ensure that enforceable source code escrow is reviewed by a lawyer specializing in such contracts
Determine the limitation recourse tolerance in the event of a breached agreement
In addition to answering these questions, the IS auditor should review the plan to ensure that there are clear guidelines and responsibilities for the declaration of disaster, the movement to the off-site facility, and the restoration of normal business operations when the disaster is over. Both the facility and the contracts should be tested, reviewed, and updated to meet the needs of the organization. All personnel associated with the BCP, particularly the implementation of the disaster-recovery site, should be trained and should participate in regular testing in the off-site facility.
The organization’s insurance coverage should take into account the actual cost of recovery and should include coverage for media damage, business interruption, and business continuity processing. Mitigating the risk and impact of a disaster or business interruption usually takes priority over transferring the risk to a third party such as an insurer, but the organization should ensure that the amount of coverage will provide for the recovery of income and equipment in the event of a disaster. The amount of coverage as well as the items covered will vary depending on whom the organization has a policy with. There are two general types of insurance: property and liability.
Property insurance can protect the organization from a wide variety of losses, including these:
Buildings
Personal property owned by the organization (tables, desks, chairs, and equipment)
Loss of income
Earthquake
Flood (usually an additional rider on the policy)
Property insurance can be structured to cover computer equipment, software, and vital records, as well as the loss of income that would result from disruptions or disasters.
A general liability policy is designed to provide coverage for the following:
Personal injury
Fire liability
Medical expenses
General liability for accidents occurring on the organization premises
The organization must ensure that all costs associated with a disaster and the recoveries are included in its insurance policies. It might be necessary to purchase additional insurance policies to extend coverage (sometimes called umbrella policies) or purchase specific insurance coverage (flood or terrorism, for example) based on the needs of the organization.
The BCP team should define key personnel within the business units and IT to implement the plan. These personnel should be a part of the planning, testing, and maintenance of the BCP. Key personnel should have alternates to function in their place, where necessary. Per ISACA, response team structures within the BCP might include the following:
Emergency action team—. These are the first responders and deal with the immediate effects of the disaster. One of their primary functions is to evacuate personnel and secure human life.
Damage-assessment team—. This team assesses the damage immediately following the disaster, to provide the estimate of time to recover.
Emergency-management team—. This team is the primary coordinator for the recovery efforts. It handles key decision making and directs recovery teams and business personnel. It also handles financial arrangement, public relations, and media inquiries.
The emergency-management team should coordinate the following activities:
Retrieving critical data from off-site storage facilities
Installing and testing systems software and applications at the system-recovery site
Identifying, purchasing, and installing hardware at the system-recovery site
Operating from the system-recovery site
Rerouting network communication traffic
Re-establishing the user/system network
Transporting users to the recovery facility
Reconstructing databases
Supplying necessary office goods, such as special forms, check stock, paper, and so on
Arranging and paying for employee relocation expenses at the recovery facility
Coordinating systems use and employee work schedules
Off-site storage team—. This team is responsible for obtaining, packaging, and shipping media and records to the recovery facilities, as well as establishing and overseeing an off-site storage schedule for information created during operations at the recovery site.
Software team—. This team is responsible for restoring system service packs, loading and testing operating systems software, and resolving system-level problems.
Applications team—. This team travels to the systems-recovery site and restores user packs and application programs on the backup system. As the recovery progresses, this team might have the responsibility of monitoring application performance and database integrity.
Security team—. This team continually monitors the security of system and communication links, resolves any security conflicts that impede the expeditious recovery of the system, and ensures the proper installation and functioning of the security software package.
Emergency operations team—. This team consists of shift operations and shift supervisors who will reside at the systems-recovery site and manage system operations during the duration of the disaster and recovery projects. Another responsibility might be coordinating hardware installation, if a hot site or other equipment-ready facility has not been designated as the recovery center.
Network-recovery team—. This team is responsible for rerouting wide-area voice and data communications traffic, re-establishing host network control and access at the system-recovery site, providing ongoing support for data communications, and overseeing communications integrity.
Communications team—. This team travels to the recovery site, where its members work in conjunction with the remote network-recovery team to establish a user/system network. This team also is responsible for soliciting and installing communication hardware at the recovery site, and working with the local exchange carriers and gateway vendors in the rerouting of local service and gateway access.
Transportation team—. This team serves as a facilities team to locate a recovery site, if one has not been predetermined, and is responsible for coordinating the transport of company employees to a distant recovery site. It also might assist in contacting employees to inform them of new work locations and scheduling and arranging employees’ lodging.
User hardware team—. This team locates and coordinates the delivery and installation of user terminals, printers, typewriters, photocopiers, and other necessary equipment. This team also offers support to the communication team and to any hardware and facilities salvage efforts.
Data preparation and records team—. This team works from a terminal that connects to the user recovery site and updates the applications database. The team also oversees additional data-entry personnel and assists in record-salvage efforts in acquiring primary documents and other input information sources
Administrative support team—. This team provides clerical support to the other teams and serves as a message center for the user-recovery site. This team also might control accounting and payroll functions, as well as ongoing facilities management.
Supplies team—. This team supports the efforts of the user hardware team by contacting vendors and coordinating logistics for an ongoing supply of necessary office and computer supplies.
Salvage team—. This team manages the relocation project. It also makes a more detailed assessment of the damage to the facilities and equipment than was performed initially, provides the emergency-management team with the information required to determine whether planning should be directed toward reconstruction or relocation, provides information necessary for filling out insurance claims, and coordinates the efforts necessary for immediate records salvage, such as restoring paper documents and electronic media.
Relocation team—. This team coordinates the process of moving from the hot site to a new location or to the restored original location. This involves relocating the information systems–processing operations, communications traffic, and user operations. This team also monitors the transition to normal service levels.
The response teams are responsible for the tasks associated with everything from evacuating personnel and securing human life, to relocating and resuming critical business functions. Each individual on the response team should have clearly defined responsibilities and documented procedures on how to perform their tasks.