Recoverability Growth Model

The recoverability growth model is a corollary to the reliability growth model, effectively demonstrating that where recoverability principles are prioritized in software, the software recovery should decrease asymptotically over time toward theorized automatic, immediate recovery. The recoverability growth curve is demonstrated in Figure 5.3, in which lower recoverability denotes longer recovery periods and higher recoverability denotes shorter recovery periods.

The recoverability growth curve often occurs in conjunction with the reliability growth curve. Thus, as SAS practitioners hone software and make it more resilient and robust, they often additionally seek to improve recovery because they learn ways to eliminate or make recovery phases more efficient. The recoverability curve is irrespective of the reliability curve, as Figure 5.4 demonstrates both the combined and individual effects of reliability and recoverability in relation to each other.

Availability is demonstrated by the percent of time that software is not in a failed state. As demonstrated in Figure 5.4, while the reliability growth curve and recoverability growth curve can occur separately over the software lifespan, the highest software availability occurs when both reliability and recoverability are maximized.

Timely

Just as software speed is often used as a proxy to represent overall software performance, timeliness of recovery often acts as a proxy for other recoverability principles. When software recovery occurs quickly or instantaneously, the assumption is that other principles must have been followed. Thus, other recoverability principles often are not prioritized unless recoverability or reliability performance metrics are not being achieved.

At a higher level, recovery is viewed as a single event that restores functionality and terminates the recovery period. The RTO, if present in requirements, specifies the time period within which the average recovery must occur. Thus, the actual MTTR achieved over time should always be less than the RTO although individual recovery periods may exceed the RTO. In more restrictive environments, the RTO is interpreted as a maximum recovery period that no single recovery period should ever exceed. In this latter sense, implementation of the RTO more closely approximates the MTD. Figure 5.5 demonstrates six software failures whose recovery periods have been measured and recorded against recoverability requirements. While the MTTR fell below the RTO, one failure did take longer to recover than the RTO, which represents a performance failure. All recoveries did occur far below the MTD that stakeholders had established.

In the case of Figure 5.5, in which the RTO was not achieved because of one performance failure, attention should shift from higher- to lower-level recovery, including discrete phases that can occur during the recovery period. For example, SAS practitioners might discover that during the single failure, the cleanup phase of recovery took longer because invalid data products had been created and had to be corrected. Thus, in seeking to deliver more consistent recovery in the future, timeliness of the SAS software could be improved by investigating and improving individual phases of recovery.

Efficient

Efficiency can be represented by the effort or resources that a person or process must expend to facilitate software recovery. For example, an inexperienced SAS practitioner who is unfamiliar with specific software would be less efficient in investigating and remedying a failure than a more experienced developer who had authored the software. Code that is difficult to read or poorly documented can also unnecessarily delay recovery because it is not intuitive and thus more difficult to maintain and modify. A failure that requires substantial cleanup of the SAS system because the code produced spurious data, metadata, or data products can also be an inefficient use of a developer's time, because that cleanup could likely have been eliminated or could have occurred automatically, were more rigorous software development practices enforced. While software failure is never a pleasant experience, through proactive design and development, developers can ensure that their time is well utilized during the recovery period.

Development time, however, is not the only resource that must be taken into account. Software itself should recover efficiently, demonstrating a competent use of system resources and execution time. If SAS code fails to recognize that an internal failure has occurred and the failure goes unnoticed for hours or days, this is an inefficient use of the SAS processor, which was effectively churning without purpose for days while producing invalid results that must be reproduced after recovery. When a program must be rerun at the close of the recovery period, efficient code should also intelligently bypass any unnecessary steps. Checkpoints can demonstrate code modules that have completed successfully prior to failure, thus enabling process flow to skip these and proceed directly to the point of failure. This technique is demonstrated in the “Recovering with Checkpoints” section later in the chapter.

Autonomous

Autonomy typically includes and supersedes automation, specifying that a system is not only free from human interaction but also able to make decisions intelligently. Fuzzy logic SAS routines can guide processes to detect failure automatically and, based on metadata collected about the failure, to recover automatically in many cases. Process metrics collected during software failure can demonstrate the type and location of exceptions or errors, especially if they can be compared with baseline metrics that demonstrate previous process success. By predicting and detecting specific error types in exception handling routines, software can often terminate gracefully, thus setting the stage for a cleaner, more efficient recovery.

Consider a SAS process that fails because a required input data set is not available when the software is run. Autonomous recovery begs the question, “What steps would a skilled SAS practitioner follow in this situation?” and recreates this logic in software. For example, when alerted to the exception of the missing data set, the software might automatically send an email to stakeholders to inform them while entering a busy-waiting loop for a number of hours to give developers the chance to rectify the missing data set. After a parameterized amount of time, the software could gracefully terminate if the data set still could not be located. However, if stakeholders were able to restore the availability of the input data set, recovery could occur quickly and efficiently without a single runtime error.

This scenario underscores the importance of autonomy and automation throughout all recovery phases. For example, the obvious bottleneck in the proposed solution is the intervention of SAS practitioners, who must first receive an email and then perform some action. A single bottleneck that relies on human interaction can cause significant delays in recovery systems that are otherwise streamlined and automated, but in many cases it's unrealistic to expect an automated process to transfer the necessary data set to its expected location, so total automation may not be achievable. Notwithstanding, even adding processes that automatically detect and communicate software failures will provide a tremendous advantage for software recovery.

Constant

Constancy or consistency reflects stability in code over time, including the manner in which software is executed and the steps taken to revive software following failure. Stable software should be dynamic enough to determine how and where to restart after a failure. Thus, rather than highlighting sections of code to be run manually or creating quick fixes to be used to jumpstart code, developers should be able to run the same code in its entirety every time—whether following a successful or failed execution. Temporary solutions that can revive and run software today may be timely but, if they introduce new defects or technical debt that must be dealt with tomorrow, this added cost is typically not worthwhile.

For example, suppose that a serialized extract-transform-load (ETL) process is executing that ingests and transforms 20 SAS data sets in series. If the process fails on the 16th data set after successfully processing 15 data sets, some SAS practitioners might be tempted to rerun the software with manual modifications that specify that the software should start at the 16th rather than the 1st data set. This type of rogue emergency maintenance is abhorred in production software and should be avoided at all cost! Another tempting but terrible idea would be to rerun the software in its entirety, thus needlessly reprocessing the first 15 data sets and obviously violating the principles that recovery should be timely and efficient.

A consistent software solution, on the other hand, might instead consult a control table to interrogate performance metrics, which would demonstrate that the first 15 data sets had been processed, thus dynamically enabling the software to begin processing on the 16th data set rather than the 1st. As a result, software can remain constant by implementing dynamic processes that effectively utilize checkpoints, thus eliminating the poor practices of rogue software modification or redundant processing. This revised solution, which dynamically assesses past successes and failures of the software, can be utilized to execute software every time it is run, regardless of whether the previous execution succeeded or failed. The benefits of consistent, stable software are demonstrated throughout chapter 17, “Stability.”

Harmless

When harmless software fails, it should not adversely affect its environment, such as by producing faulty, incomplete, corrupt, or otherwise invalid data, metadata, data products, or output. Harmless code should automatically detect a failure when it occurs and handle it in a predictable, responsible manner, because only through predictability comes the assurance that dependent processes and data products are not negatively affected. In most cases, it is far more desirable to have data products that are unavailable or delayed (due to detected failure) than those that are dead wrong (due to undetected failure).

Not only failure but also recovery should be harmless, so the environment should not be altered unfavorably in an attempt to restore functionality. For example, if a developer modifies SAS library names in a production environment as an expedient solution to rerun code that has failed (because it erroneously referenced library names in the development environment), this could cause unrelated processes to fail. Another scenario often occurs in which software fails because of an exception or error within a macro shared by the development team. The temptation may exist to alter the macro so that it works in the current software that is failing, but if this modified macro is reused in other production software, the change cannot be made until the implications of the proposed modification are understood. A software reuse library, demonstrated in the “Reuse Library” section in chapter 18, “Reusability,” can facilitate an awareness of code use and reuse and ultimately backward compatibility when maintenance is required.

Stop Program

Failure comes in all shapes and sizes but does not necessarily denote program termination or even that runtime errors were produced. Failure can occur not only when software produces runtime errors or terminates abruptly, but also when it enters a deadlock or infinite loop, or continues to execute albeit producing invalid data, data products, or other output. A common goal of exception handling routines is to channelize software outcome toward two distinct paths, the happy trail and the fail-safe path.

The happy trail (or happy path) is the journey that software takes from execution to successful completion and delivery of full business value. When exceptions or errors are encountered, however, robust software aims to get back on the happy trail toward restoration of full or partial functionality. In other words, something bad happens during execution, but the software limps along or is fully restored, ultimately delivering either full or partial business value. Software that lacks robustness will still have an underlying happy trail, but once exceptions occur, no methods exist to rejoin that trail, so the software can only fail.

The fail-safe path is the alternative path in which software fails gracefully due to exceptions or errors that were encountered. Functionality halts and no or minimal business value is delivered, but the environment is undamaged and remains secure. An example of unsecure termination would be abruptly and manually aborting SAS software because invalid output had been detected by analysts. Secure termination, on the other hand, might automatically detect the abnormalities in output, update a control table to reflect the exception or error, and instruct SAS to terminate. Thus, automatic software termination must always begin with automatic failure detection.

Although production software should fail gracefully when errors are encountered, if an unexpected error is encountered for the first time, software may fail in an unsafe or unpredictable manner. In these cases, if software is still executing but errors have been detected, it must be terminated manually. For example, if an ETL program that performs daily data ingestion requires tables to have 40 fields, but on Monday it encounters a table that has only 37 fields, the process flow should be robust enough to detect this exception automatically. Business rules would direct the exception handling, but might include immediate program termination with notification to stakeholders, as well as preventing dependent processes from executing. If the system is not robust and does not detect the 37-field discrepancy, subsequent code could continue to execute and create invalid results, even if no runtime errors were produced.

Timely, efficient, and autonomous termination can often be achieved through exception handling routines that detect exceptions and runtime errors and provide graceful program termination. For example, the following SAS output demonstrates an error that is produced when the file Ghost does not exist:

data temp;
   set ghost;
ERROR: File WORK.GHOST.DATA does not exist.
run;
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.TEMP may be incomplete. When this step was stopped there were 0 observations and 0 variables.
WARNING: data set WORK.TEMP was not replaced because this step was stopped.
%put &SYSCC;
1012

Testing the automatic macro variable &SYSCC after each procedure, DATA step, or other module offers one way to determine programmatically whether a runtime error occurred and can be utilized to alter program flow dynamically thereafter. The use of SAS automatic macro variables for use in exception detection and handling is described extensively in chapter 3, “Communication,” and in the “Exception Handling” section in chapter 6, “Robustness.”

Software failures, however, are not always signaled by runtime errors. The 37-field data set described earlier was structurally unsound and would produce invalid results yet possibly yield no obvious runtime errors. As another example, if the following code is run after the previous code, it runs without producing a runtime error but is still considered to have failed because an empty data set is produced:

data temp2;
   set temp;
run;
NOTE: There were 0 observations read from the data set WORK.TEMP.
NOTE: The data set WORK.TEMP2 has 0 observations and 0 variables.

It turns out that despite encountering an error, the first DATA step creates the Temp data set even though it has zero observations and zero variables. Thus, the second DATA step executes without warning or runtime error, despite having invalid input. In testing for conditions that should direct a program to abort, software must examine more than warning and error codes and must validate that business rules and prerequisite conditions are met. The following code first tests to ensure that the data set contains at least one observation and, if that requirement is not met, the %RETURN statement causes the macro to abort:

%macro validate(dsn=);
%global nobs;
%let nobs=;
data _null_;
   call symput('nobs',nobs);
   stop;
   set &dsn nobs=nobs;
run;
%mend;
* inside some parent macro;
%validate(dsn=work.temp);
%if &nobs=0 %then %return;
* else continue processing;

Despite diligence in software exception management, on occasion software will still need to be terminated manually. Terminating software manually reinforces the reality that software can fail inexplicably at any line of code. The hapless developer who trips over a cable and disconnects his machine from the server could do so while any line of code is executing. Recoverability requires developers to espouse this mind-set and ask themselves, after every line of code, “What will happen if my code fails here?” This mind-set reflects that code should always fail safe, in a harmless, predictable manner. This aspect is discussed more fully in the “Cleanup” section later in this chapter, because code that fails safe and is harmless will not require cleanup, thus avoiding this sometimes agonizing aspect of the recovery period.

Investigation

Where program termination has resulted from a scheduled outage due to maintenance or other program modification, this step won't be necessary because failure was planned. But in cases in which unplanned failure has occurred in production software, stakeholders will be clamoring not only for immediate restoration but also for answers. From a risk management perspective, higher-level stakeholders are often not concerned with great technical detail, but rather with more substantive facts surrounding the failure, such as “What caused this failure?” and “Will it happen again?” A best practice is to record these and other qualitative information about the failure in the failure log, as discussed in the “Qualitative Measures of Failure” section in chapter 4, “Reliability.”

Log files are a recommended best practice for production software because they can help validate program success through a baseline (i.e., clean) log and facilitate the identification of runtime or logic errors. When the failure source is captured in the log file and through continuous quality improvement (CQI), developers are able to refactor code to improve robustness, eliminating one failure at a time from occurring in the future, as depicted in the “Reliability Growth Model” section in chapter 4, “Reliability.” In other cases, the log file may exist but may not contain sufficient information to debug code. This also occurs when errors in business rules or logic may not be immediately perceived, resulting in the deletion of relevant log files before they can be analyzed. When this occurs, the code may require extensive debugging and test runs to reproduce and recapture the failure with sufficient information to understand the underlying defect or error.

Maintenance of a risk register is another best practice for production software because it describes known defects, faulty logic, and other software vulnerabilities that can potentially cause program failure. After encountering a failure, SAS practitioners can turn to the risk register to investigate whether associated threats and vulnerabilities had been identified for the failure and, in many cases, whether maintenance or solutions had been proposed. This is not to suggest that action will be taken once the source of failure is identified, but elucidation of the vulnerability is critical to assessing risk in reliable software. Use of the risk register is discussed in “The Risk Register” section in chapter 1, “Introduction.”

Cleanup

After investigation of a failure, the next step is either to rerun the software or perform maintenance if necessary. Especially in cases in which no defect exists and software failed due to an external threat such as a power outage, software should be able to be restarted immediately without further intervention. This pushbutton simplicity can be thwarted, however, when failures produce wreckage in the form of invalid data products—including data sets, metadata, control tables, reports, or other output. The principle that recovery be harmless dictates that carnage not be produced by production software; thus the site of failed software should not resemble an accident scene, with bits of bumpers, broken glass, and the fetid stench of motor oil and transmission fluid seeping into the asphalt.

Cleanup is typically required when some module or process started and did not complete, but left the landscape nevertheless altered. Some examples include:

• A data set is repeatedly modified through successive processes. After the failed process, the original data set is no longer available in its original format because some (but not all) transformations have occurred.
• A data set was locked for use with a shared or exclusive lock, after which the process failed before the data set could be unlocked. The data set may be unavailable for future use until the SAS session is terminated.
• The structure or content of a directory is modified, after which a failure occurs. For example, if raw data files were moved from an Original folder to a Processed folder, they may need to be moved back if the ingestion process was later demonstrated to have failed.
• A control table was updated to reflect that a process started but, because the process failed, the control table may never have been updated with this information. In many cases, SAS practitioners must manually edit the control table to remove these data, which can be risky when control tables contain months of precious, historical performance metrics and other metadata.

The first example often occurs in serialized ETL processes, in which each DATA step or procedure is dependent on the previous one. A DATA step may undergo successive transformations that, to save disk space or avoid complexity, are performed on the same data set without modifying its name. In the following example, the Final data set seems to indicate some finality, but instead is modified several times. Thus, if the code fails, it may be difficult to determine which transformations have occurred successfully (and should not be repeated during recovery). Moreover, where transformations permanently delete, modify, or overwrite data, it may be impossible to recreate the initial data set if necessary:

data final;
   length char1 $15;
   char1='taco';
run;
data final;
   set final;
   length char2 $15;
   char2='chimichanga';
run;
data final;
   set final;
   length char3 $15;
   if char1='taco' then char3='flauta'; * theoretical failure    occurs here;
run;

Cleanup can also be required when explicit locks are not cleared because software fails before this can be done. Explicit locks are beneficial because they can ensure that a data set will be available before software attempts to utilize it. When an exception is encountered in which the data set is either in use or missing, the process can enter a busy-waiting cycle to wait for access or it can fail gracefully, thus avoiding runtime errors. The LOCK statement explicitly invokes an exclusive lock and was historically used in busy-waiting cycles, but its use has been deprecated (outside of the SAS/SHARE environment), as discussed in the “&SYSLCKRC” section in chapter 3, “Communications.” Rather, FOPEN can be used to secure an exclusive lock on a file or data set, and OPEN to secure a shared lock. These functions, however, require the FCLOSE and CLOSE functions, respectively, to ensure file streams are closed, as demonstrated in the “Closing Data Streams” section in chapter 11, “Security.”

Another example of cleanup describes changes in folder structure or content that often occur as part of an ETL process. Consider an ETL process that I once inherited from another SAS practitioner. Raw text files were continuously deposited into a folder Inbox and, once a day, the ETL process would first create a folder named with the current date, then copy all accumulated text files from Inbox to the new folder, then ingest and process the files. The software was fairly reliable but, when it did fail, it failed miserably and required extensive cleanup.

When the text files were copied from Inbox to the respective daily folder, this process created an index that was later used to ingest each file. So, if the process failed sometime after files had been copied during ingestion, when the software was restarted, no files would be in the Inbox, so no new index would be created. Thus, to restart the software after a failure, the files had to be moved back to the Inbox, the index deleted, and a separate control table updated to reflect that the process was being rerun. While moving data sets or other files on a server can be a regular part of data maintenance and should be automated to the extent possible, SAS practitioners must also consider the implications of whether those actions and automation could cause later cleanup if software were to fail.

In all cases, SAS practitioners should strive to design and develop processes that require no cleanup, which not only can be costly but also can introduce the likelihood of further errors. Having to copy or rename files, modify data sets, edit control tables, or unlock data sets manually is risky business and should be avoided on production software.

Internal Issues

Once software has failed and stopped, the cause and potential remedies have been investigated, and any necessary cleanup has occurred, the next step is often to rerun the software. For example, this occurs when the software failed because of a power or server outage and no software maintenance is necessary. In the most straightforward failures, internal issues do not exist, and this phase is skipped entirely.

But what happens when the power fails and software isn't able to start immediately? Perhaps extensive cleanup of the landscape is required, or numerous software processes that had already completed need to be rerun, causing inefficiency. These all represent internal issues that should be corrected if recoverability has been prioritized, despite the actual failure occurring due to external causes. Thus, internal issues primarily describe corrective, emergency, or adaptive maintenance that must be performed on software either to restore functionality or to improve recoverability when future failures occur.

Some internal issues may not require software maintenance but may require SAS practitioners to resolve other issues in their environment. For example, if ETL software fails because a data set is locked by another process, the second process may need to be stopped (or waited for) so that the ETL software can be rerun after the data set is made available. In a similar example, memory-intensive SAS software can cause an out-of-memory failure, sometimes in the software itself and sometimes in other SAS software running concurrently on the server. Thus, when memory failures occur, other processes may need to be stopped (or waited for) so that the primary software can be restored and run with sufficient memory.

But if vulnerabilities or defects were identified in the code, or if developers or other stakeholders have decided that the recoverability or performance needs to be improved through maintenance, then the software itself must be modified in this phase. Maintenance types such as adaptive, perfective, corrective, and emergency are described in chapter 13, “Maintainability,” and by embracing maintainability principles in software design, maintenance can be performed efficiently after a failure, minimizing the recovery period.

External Issues

External issues reflect anything that stands between SAS practitioners and the recovery of their software—anything over which they have no control. The cause of failure may have been external, such as a failed connection to an external database administered by a third party. In this case, SAS practitioners would begin making phone calls to attempt to ascertain the cause of the failure and its expected duration. In many cases when an external cause is identified during investigation, a development team can only sit around and wait for the external circumstance to be rectified.

So what else can SAS practitioners do once software has crashed while they're twiddling their thumbs waiting for someone to restore a data stream? My team once lost a critical data connection to Afghanistan, and team members kept asking one of our developers, “Is the connection up yet? Do we have data?” And each time he was asked, the developer would disappear to his desk and rerun the software, producing hundreds of runtime errors that demonstrated not only the single exception—the broken data stream—but also cascading failures resultant from a total lack of exception handling. As it turns out, when external issues are preventing software from executing, this is the perfect time to build data connection listeners and other quality controls that test negative environmental states.

A negative state could represent a missing data set, a locked data set, a broken data stream, or other types of exceptions. Exception handling frameworks can be constructed to identify theoretical exceptions, but they can be difficult to test without recreating the exception in a test environment. For example, the following code tests to determine if a shared lock can be gained for a data set; only if the lock is achieved will the MEANS procedure be executed. This exception handling avoids the runtime error that occurs if another process or user has the data set exclusively locked when the MEANS procedure attempts to gain access:

data perm.original;
   length char1 $10;
run;
%macro freq(dsn= /* data set in LIB.DSN or DSN format */,
   var= /* single variable for frequency */);
%local dsid;
%local close;
%let dsid=%sysfunc(open(&dsn));
%if %eval(&dsid>0) %then %do;
   proc freq data=&dsn;
      tables &var;
   run;
   %let close=%sysfunc(close(&dsid));
   %end;
%mend;
%freq(dsn=perm.original, var=char1);

This exception handling will prevent failures from occurring from a locked or missing data set but, because exception handling is only as reliable as the rigor with which it has been tested, each of these states must be tested individually. A missing data set is easy to test—just delete Original and invoke the %FREQ macro as indicated, and the FREQ procedure will be bypassed. To test how exception handling detects and handles shared and exclusive file locks on Original, a separate session of SAS must be initiated in which Original is first locked with a shared lock and subsequently with an exclusive lock while the %FREQ macro is run from the first SAS session. This process is more complicated, but it is well worth the peace of mind to know that the framework handles negative states as expected.

However, exception handling designed to detect external exceptions or errors can be more difficult to test because SAS practitioners inherently don't have control over external data streams to test negative states. When a data stream is down—and down long enough to write and test some code—this presents the perfect opportunity to ensure that future, similar exceptions will be caught and handled programmatically and automatically. When our Afghanistan data stream was down for several hours, we built a robust process that could detect the exception—the failed data stream—and enter a busy-waiting cycle that would repeatedly retest every few minutes until the stream was recovered. Moreover, we directed these test results to a dashboard so all stakeholders could monitor data ingestion processes.

The refurbished solution was superior for several reasons and drastically improved recoverability of the software. By detecting the exception, the exception handling framework prevented cascading failures and invalid data sets that would have had to be deleted. The solution enabled the team to be more efficient during recovery because no one had to retest the software every time a customer asked for an update—this had been automated. Moreover, the recovery itself was made more efficient because, when the connection eventually was restored, the busy-waiting cycle identified this and ran the software only a few minutes after the connection was restored. And, finally, the solution promulgated the connection status—failed or active—to all stakeholders through the dashboard, thus helping us to more quickly identify and resolve similar negative states in the future.

This example demonstrates dichotomous states in which the external connection is either active or failed, and in which failure denotes total loss of business value and an active connection denotes full business value. But external issues often lack this dichotomy, especially where partial business value can still be delivered. For example, a similar database connection linked my team to a third-party Oracle database maintained overseas. Sometimes the entire database would be down due to external failure; in other cases, only some of the Oracle tables were unreachable. At this point, we had to make the business decision of whether to reject the entire database or pursue partial business value—we opted to take what we could get.

In this scenario, partial business value was deemed better than no business value, but we had to manipulate our software to detect and respond to the exception of individual missing tables. In so doing, we took a SAS process that had been consummately failing when only one table was missing and developed a solution that identified and bypassed missing tables and that conveyed available data to subsequent analytic reporting. For missing tables, the software essentially entered a busy-waiting cycle (similar to the prior data stream example) in which it tested the availability of those specific missing tables until they could be ingested. In delivering this solution, we ensured that business value was maximized despite the occurrence of external failures beyond our control. Furthermore, we received kudos from the data stewards because, in many cases, we were able to inform them that their database or tables were down before they were even aware!

Rerun Program

Once any necessary modifications to code, hardware, or other infrastructure have been made, the SAS software should be tested, validated, and executed. Because the cause of the failure was identified and remedied (or at least evaluated and included in the risk register), in an ideal scenario, the developer hits Submit or schedules the batch job and confidently walks away. Execution often is less straightforward, however, and unfortunately can involve highlighting bits and pieces of code to run in an unrepeatable fashion following a failure. As described previously in the “Constant” section, quick fixes should be avoided and the software that recovers best is the software that can be rerun in its entirety and autonomously skip processes that do not need to be rerun.

When SAS software fails in an end-user development environment, developers often make a quick fix, reattempt the offending module, and then run the remaining code from that point. End-user developers are well placed to make these modifications on the fly because they are, by definition, the primary users of the software. While this “code-and-fix” mentality is appropriate for end-user development, production software should never be maintained in a code-and-fix fashion. A more robust solution is to engineer SAS software that intelligently determines where it should begin executing, thus enabling both autonomous and efficient recovery. In addition to facilitating recovery after failure, this design further facilitates the recurrent software recoveries necessary during testing and development phases. In other words, the extra development effort expended to facilitate autonomous, efficient recovery will more than pay for itself over the course of the software lifespan.

To continue the scenario from the prior “Constant” section, consider that you have an ETL process that ingests and transforms 20 data sets in series. A failure occurs after the 15th data set has completed processing. SAS practitioners in end-user development environments might be tempted to modify the code so that when the software recovers, only the last five data sets are specified to be processed. This short-term solution should be avoided, especially where it would involve having to remodify the software afterward so it could again process all 20 data sets in the future.

A more autonomous yet less timely solution would be to run the entire process again, thus also redundantly processing the first 15 data sets. This duplication of effort would be neither timely nor efficient (for the SAS system), but it would at least be efficient for SAS practitioners because they wouldn't be stuck babysitting code during the recovery period. However, if ultimate business value is conveyed through analytic reporting that can be produced only once all 20 data sets have been processed, it would represent a tremendous inefficiency for the business analysts or other stakeholders who might be waiting on the other end of the SAS software for data products so that they can generate their reporting. Clearly a solution is required that is both efficient and autonomous.

A superior solution is to design modular code that intelligently identifies where it has failed and succeeded, and records process metrics in a control table that can be used to drive processing both under normal circumstances as well as under exceptional circumstances, including during and following catastrophic failure. By implementing checkpoints (demonstrated in the following section), the software can start where it needs to—where the failure occurred—rather than inefficiently reprocessing steps that had completed successfully. Through these best practices, timely, efficient, autonomous, constant, and harmless recovery can be achieved. Other benefits of control tables are demonstrated in the “Control Tables” section in chapter 3, “Communication.”

Requiring Recoverability

Recoverability is less commonly prioritized in software requirements than reliability even though the recovery period directly detracts from software availability, the key reliability metric. Stakeholders too often want to increase reliability by preventing failures alone rather than by also reducing the impact that each failure has on the system. Incorporating recoverability metrics into software planning and design is beneficial because it enables SAS practitioners to further understand the quality that their software must demonstrate. Moreover, when failures do occur, all stakeholders have common benchmarks by which to assess whether recovery is occurring as expected or whether it is delayed, chaotic, or otherwise failing to perform to standards.

While the five principles of recoverability are closely intertwined, timeliness is the most quantifiable metric; it has been mentioned that timeliness often acts as a proxy for other principles. By specifying that software recovery additionally must be efficient, autonomous, constant, and harmless, software requirements set the stage for a timely recovery. In technical requirements, it's common to simply state the MTTR, RTO, and possibly MTD that are required of software performance. These terms are defined in the “Defining Recoverability” section earlier in this chapter and further described in the “Timely” section.

Measuring Recoverability

Consider that a SAS process runs every hour on the hour 24×7 for March and April but has experienced five failures, each of which is recorded in the failure log. Table 5.1 presents this failure log in an extremely abbreviated format, reprising the failure log demonstrated in the “Measuring Reliability” section in chapter 4, “Reliability.” Please visit that section to understand the difference between reliability and recoverability metrics.

The “Redefining Failure” and “Redefining Recovery” sections earlier in the chapter delineate a host of issues involving the interpretation of these two terms; however, for the purposes of this demonstration, failure and recovery are represented in the failure log but not further defined. Thus, the failure occurring on March 16 incurred a recovery period of 18 hours, with subsequent recovery periods including 20, 1, 1, and 4 hours. Note that unless otherwise specified, the cause of the failure—internal or external—is irrelevant, as is severity of the failure, so every failure and its corresponding recovery contributes to recoverability metrics. In this general sense, the MTTR is (18 + 20 + 1 + 1 +4) / 5 or 8.8 hours.

Well, this doesn't seem too fair. The first 18-hour outage was caused by a power failure over which the data analytic team likely had no control. For this reason, team- and organizational-level documentation such as SLAs often explicitly state the software or infrastructure for which a team will have responsibility—including incurring risk of failure—as well as external infrastructure for which they will not be held responsible. Thus, in a slightly modified example, if the team explicitly is not held responsible for power or network outages, then the first and third items in the failure log should not be included in recoverability metrics. In this example, the MTTR is (20 + 1 + 4) / 3 or 8.3 hours, only negligibly better than the previous example.

The first and second failures in the log also illustrate an important point: the recovery period reflects the duration of recovery but not necessarily the effort involved. When an externally caused failure occurs, the majority of the recovery period can be the team performing unrelated tasks while they wait for power to return or a system to come back online. Even when SAS practitioners work diligently to repair a failure like the one experienced on March 18, it's unlikely that they literally worked around the clock until 10 AM to recover the software. More likely, they stayed late but still made it home in time for dinner or at least to sleep. Yet, because they had not yet successfully recovered the software, the time which they slept counted for recovery metric purposes. Thus, the critical point to understand is that when stakeholders are discussing whether and how to implement recoverability metrics such as RTO or MTD, realistic estimates and thresholds must be generated that take into account variability such as software that remains in a failed state overnight.