Avoiding the Quality Trap

Quality gets bandied about so frequently in everyday conversation that it's important to differentiate its very specific definition and role within software development. In general, when you speak of quality in isolation such as that's a quality alpaca, you're expressing appreciation for product or service excellence. And, similarly, when you discover a high-quality Thai restaurant, you want to dine there often because the drunken noodle is delicious! In this sense, quality can represent an isolated assessment of the food, or it could represent a comparison of the high-quality Thai food to other Thai restaurants whose food was not exceptional. Thus, in assessing or describing quality, a comparison may intrinsically exist, but may be inferred rather than specified outright.

As a software user, I find this comparative use of quality is common. For example, I use Gmail because it offers greater functionality and performance than other email applications. In terms of functionality, I appreciate the threaded emails, overall layout, and ability to thoroughly customize the environment. In terms of performance, the high-availability service is extremely reliable and I trust the security of the Gmail servers and infrastructure. But in exclaiming that Gmail is high quality, I'm inherently making a comparison between this email application and email applications that I've previously used that offered less functionality or performance. Thus, a software user often determines quality based on his needs and his requirements and possibly a comparison of how well one software application meets those needs as compared to another.

As a software developer, however, I recognize that software quality is consistently defined by organizations like ISO and IEEE, which specify that quality must always be assessed against needs or requirements, rather than in isolation or comparatively to other software. These definitions are in line with the view espoused in product and project management literature: that quality represents “the degree to which a set of inherent characteristics fulfill requirements.”⁴ Without knowledge of the intent with which software was created, the software function and performance can be described, but it's impossible to determine software quality—this is the quality trap to which we fall victim, due to the chasm between generalized and industry-specific usage and definitions of quality.

So if the term quality carries so much baggage, how are you supposed to assess software quality or even discuss it? How do you tell a coworker that his software sucks? Software quality models are recognized as an industry standard because of the organized, standardized nomenclature that fully describes software performance. The primary objective of this text is to demonstrate the ISO software product quality model, its benefits, its limitations, and its successful implementation into SAS data analytic development and into the lexicon with which we discuss software and software requirements.

A New Quality Vocabulary

To begin a rudimentary assessment of software quality, imagine the following SAS data set exists:

data sample;
   length char1 $20 char2 $20;
   char1="I love SAS";
   char2="SAS loves me";
run;

You're now asked to assess the quality of the following SAS code, which simulates a much larger data transformation module within extract-transform-load (ETL) or other software:

data uppersample;
   set sample;
   char1=upcase(char1);
   char2=upcase(char2);
run;

So is the code high quality, low quality, or somewhere in the middle? In other words, would you rehire the SAS practitioner who wrote it, or fire him for being a screwup? Therein lies the quality trap. Given ISO and IEEE quality definitions, it's impossible to know whether this represents high- or low-quality software without first understanding the objective, needs, and requirements that spawned it.

You subsequently find an email from a manager requesting that the previous code be developed:

At this point, you can understand the ultimate project need and can assess that the code is high quality because it meets all requirements and was developed quickly. The only functional requirement given was capitalization, and the only performance requirement that can be inferred is speed of processing, since the manager needed the code written and executed quickly. It's not fancy, but it works and delivers the business value specified by the customer.

But now imagine that the manager instead had sent the following alternative email to request the software:

This alternative requirements definition specifies software with identical functionality (at least when applied to the Sample data set), but is more dynamic and complex. The manager requested a modular solution, so a SAS macro probably should be created. Moreover, given that the macro requires that the data set and variable names be dynamic, the module should be able to be reused to perform the same function in other contexts and on future data sets. This would be of tremendous advantage if the same team later required hundreds of variables to be capitalized in a larger data set, which would take much longer to code manually.

Given the alternative more complex requirements, the original transformation code developed could be assessed to be of poor quality. It meets the ultimate functional objective (all variables are capitalized), but lacks performance requirements that specified it be modular, dynamic, and reusable. The SAS solution that does meet these additional performance requirements is demonstrated in the “Reusability Principles” section in chapter 18, “Reusability.” The increased code complexity also demonstrates the correlation between the inclusion of quality characteristics and increased complexity and development time, an inherent tradeoff in software development.

Dimensions of Quality

Given the specific use of quality within software development circles, the need still arises to describe software that exhibits a high (or low) degree of quality characteristics (i.e., performance requirements), irrespective of software needs or requirements. For example, back to the original question: If I'm unaware of the email that the manager sent the developer prescribing software requirements, how do I describe the quality of the code in a vacuum?

If backed into a corner, I might describe the code as relatively low quality, essentially making a comparison between this code and theoretical code that could have been written having the same functionality. More likely, I might state that the code is not production software, essentially evaluating the code against theoretical requirements that would likely exist if the code underpinned critical infrastructure, had dependencies, had multiple users, or was expected to have a long lifespan. However, a tremendous benefit of embracing a software quality model is that you can discuss the performance of software without dragging poor old quality into the fray. For example, I can instead say that the code lacks stability, reusability, and modularity—which concisely evaluates the software irrespective of needs and requirements.

The lexicon established through the adoption of software quality models benefits development environments because the derived vocabulary is essential to software performance evaluation. Only with unambiguous technical requirements that implement consistent nomenclature can quality be incorporated and later measured in software. It's always more precise and useful to state that software “lacked modularity and reusability” than to vaguely assert only that it “lacked quality.” Even when stakeholders choose not to implement dimensions of software quality in specific products, they will have gained a recognized model and vocabulary about which to discuss quality and consider its inclusion.

Data Quality and Data Product Quality

Within SAS literature, when quality is referenced, it nearly always describes data quality and, to a lesser extent, the quality of data products such as analytic reports. In fact, in data analytic development, quality virtually never describes software or code quality. This contrasts sharply with traditional software development literature, which describes characteristics of software quality and the methods through which that quality can be designed and developed in software.

This shift in quality focus is somewhat expected, given that data analytic development environments usually consider a data product to be the ultimate product and source of business value, as contrasted with traditional software development environments in which software itself is the ultimate product. The misfortune is that by allowing data quality and data product quality to eclipse (or replace) code quality, data analytic development environments unnecessarily fail to incorporate performance requirements throughout the SDLC. Thus, code quality takes a backseat to data quality during requirements definitions, design, development, testing, acceptance, and operation.

If the software that undergirds data products lacks quality, though, how convincingly can those data products really be shown to demonstrate quality? To illustrate this all-too-common paradox, I'll borrow an analogy from a friend who, while working for the Department of Defense (DoD), encountered a three-star general who was adamant that some trendlines in a SAS-produced analytic report were “not purple enough.” He was focused on the quality of the data product, to the total exclusion of interest in the quality of the underlying software. Was the software reliable? Was it robust, or could it be toppled by known vulnerabilities? Was it extensible in that it could be easily modified and repurposed if the general had a tactical need that required subtle variation of the analysis? No, none of these questions about code quality were asked, only “Can you make this more purple?!”

This text doesn't aim to diminish the importance of data quality and data product quality. Rather, it places code quality on par with data quality, with the understanding that any house built upon a poor foundation will fail.

External Software Quality

External software quality is “the degree to which a software product enables the behavior of a system to satisfy stated and implied needs when the system including the software is used under specified conditions.”⁶ External quality characteristics are those that can be observed by executing software. For example, software is shown to be reliable because it does not fail. Software is shown to be scalable because it is able to process big data effectively without failure or inefficiency.

External software quality is sometimes referred to as the black-box approach to quality, because the assessment of quality is made through examination of software execution alone, rather than inspection of the code. Thus, a black-box view of software efficiency can assess metrics such as execution time or resource utilization, but cannot determine whether SAS technical best practices that support efficient execution were implemented. Black-box testing is discussed in the “Functionality Testing” section in chapter 16, “Testability.”

In traditional software applications, users have no access to the underlying code and are only able to assess external software quality through a black-box approach. Thus, while internal software quality characteristics are important to building solid, enduring code that will be more easily maintained, only external software quality characteristics can be demonstrated to customers, users, and other stakeholders who may not have access to code. Because SAS software users often do have access to underlying code, the distinctions between black- and white-box testing (that is, testing internal software quality, which is further discussed in a later section) are less significant in SAS software development than in other languages.

Dynamic Performance Requirements

A performance requirement is “the measurable criterion that identifies a quality attribute of a function or how well a functional requirement must be accomplished.”⁹ Dynamic performance requirements are performance attributes that can be observed during software execution. In other words, they represent all characteristics of external software quality except functionality, as demonstrated in Figure 2.3. Dynamic refers to the fact that software must be executing (i.e., in motion) to be assessed, whereas static performance attributes must be assessed through code inspection when software is at rest.

Throughout this text, performance requirements are sometimes referred to as performance attributes, especially where the use is intended to demonstrate a quality characteristic that may not be required by specific software. For example, if a SAS practitioner is considering implementing scalability and security principles but decides to omit security from a software plan, security would be referenced as a performance attribute—not a performance requirement—to avoid confusion since the quality characteristic was not actually required by or implemented into the software. In general, however, the terms are interchangeable.

Because dynamic performance requirements can be observed by all stakeholders, they are traditionally more valued than static requirements, and are thus more readily prioritized into software design and requirements. For example, a customer can more easily comprehend the benefits of faster software because speed is measurable. However, unless the same customer has a background in software development, he may have a more difficult time comprehending the benefits of incorporating modularity or testability into software because these attributes cannot be directly measured or their effects observed during execution.

Internal Software Quality

Internal software quality is defined as the “degree to which a set of static attributes of a software product satisfy stated and implied needs when the software product is used under specified conditions.”¹⁰ These characteristics can only be observed through static examination of code and, sometimes in the case of readability, through additional software documentation that may exist. Testability sometimes can also be assessed through a formalized test plan and test cases, as discussed in chapter 16, “Testability.”

Internal software quality is sometimes referred to as the white-box (or glass-box) approach to quality, because the assessment of quality is made through code inspection rather than execution. A white-box view of software reusability assesses to what extent software can be reused based on reusability principles, but requires that code be reviewed either manually or through third-party software that parses code. Because users have no access to underlying code in traditional software applications that are encrypted, they have no way to assess internal software quality; if they lack technical experience, they may even have no awareness of its concepts.

Commingled Quality

The organization of individual dimensions of software quality is straightforward; however, placement of quality dimensions within superordinate structures (such as quality models) can become complex due to nomenclature variations. For example, external software quality comprises both functional and performance requirements, a dichotomy commonly made in software literature. This is reflected in the ISO software product quality model, which subsumes functional suitability (i.e., functionality) under software product quality.¹¹

Other quality models, however, include performance characteristics but omit functionality. For example, many definitions of the iron triangle—the nexus of project scope, schedule, and cost—define scope as having separate quality and functional components, thus intimating a model in which quality omits functionality. As one example of this somewhat divergent view of quality, the International Institute of Business Analysis in its Guide to the Business Analysis Body of Knowledge^® (BABOK Guide^®) distinguishes quality from functionality in its requirements definitions:

• “Functional requirements: describe the capabilities that a solution must have in terms of the behavior and information that the solution will manage.”¹²
• “Non-functional requirements or quality of service requirements: do not relate directly to the behavior of functionality of the solution, but rather describe conditions under which a solution must remain effective or qualities that a solution must have.”¹³

More quality model commingling occurs because software performance requirements comprise both dynamic and static performance requirements, with the former representing external software quality characteristics (with functionality omitted) and the latter representing internal software quality characteristics. Because this text focuses on software performance and excludes functionality (in assuming that functional requirements have been attained in all scenarios), the quality structure consistently referenced throughout this text reflects the dichotomy of dynamic versus static performance.

These commingled quality constructs are demonstrated in Figure 2.3, which highlights the roles of functionality, reliability, and maintainability through these various interpretations of quality. Whether functionality is a component of quality, or functionality and quality are both components of scope, one thing is clear—software functionality and software performance do compete for resources in a struggle to be prioritized into software product requirements. This and other tradeoffs of quality are discussed in the following section.

The Cost of Quality

As demonstrated in Figure 2.3, inherent trade-offs exist whether developing software or other products, often referred to as constraints. The PMBOK^® Guide includes quality and five other constructs as the principal constraints to projects, but states that these should not be considered to be all-inclusive.¹⁴ As a constraint, quality competes against other constructs for value and prioritization within software projects during planning and design. In fact, when developers omit performance from software, look at all the amazing things they can get instead!

1. Increased Scope Without the necessity to code performance requirements into software, developers instead can focus on additional functionality. For example, rather than making SAS software robust to failure, additional analytic reports could be developed.
2. Schedule The inclusion of performance requirements almost always makes code more lengthy and complex; as a result, designing, developing, and testing phases each will be lengthened. Without performance prioritized, the customer and other stakeholders can receive their software much faster.
3. Budget Because performance requirements take time to implement, and because time is money, project budgets typically increase with the increase in performance requirements. By omitting performance from software, a development team can either reduce budget or focus that money toward other priorities. Happy hour, anyone?
4. Resources Project resources include not only personnel but also maintaining the heat and power. By sacrificing performance requirements, fewer developers may be required to work on a project to achieve the same functionality so fewer resources will be consumed.
5. Risk Risk is not a benefit but rather the often-unintended consequence when performance is not prioritized. Because many performance requirements are designed to mitigate or eliminate vulnerabilities in software, risk typically increases as performance requirements are devalued or omitted.

While constraint implies a limiting or restrictive effect, constraints can also be viewed from the opportunity cost perspective. For example, developers can implement performance requirements into software, or, if they choose not to, they can produce quicker software, cheaper software, or software with increased functionality. To assess an opportunity, however, its value must be known. Thus, an objective of this text is to familiarize developers with the benefits of incorporating performance within software so that the opportunity cost of quality truly can be gauged. For example, if a stakeholder has decided to implement additional functionality (thus excluding additional performance), the stakeholder at least will have understood the value of the performance that could have been added, and thus will have made an informed decision based on the opportunity cost of that performance. Only when stakeholders understand the value of quality and the risks of the lack of quality can quality be measured against other project constraints such schedule, scope, or budget.

Establishing the value of quality can be difficult, especially where stakeholders disagree about the prioritization of constructs within a software development project. Sponsors who are funding software development will want it done inexpensively while users waiting for the software will want it developed quickly. Some developers may be more focused on the function of software and less concerned with performance attributes while other SAS practitioners may insist that the software be reliable, robust, maintainable, and modular. Thus, establishing consensus on what characteristics contribute to software quality and how to value that quality (against other project constructs) can be insurmountable. Nevertheless, stakeholders—including developers and non-developers alike—will be better positioned to discuss the inclusion of quality characteristics when they can utilize shared terminology demonstrated within the software product quality model.

Requiring Quality

Everything about quality begins and ends with requirements. Requirements specify the technical objectives that software must achieve, guiding software through design and development. Whether those same requirements have been achieved is determined through testing that validates software completion and demonstrates intended quality.

In chapters that introduce dynamic performance requirements, a recurring Requiring section describes how technical requirements for the respective quality characteristic can be incorporated into requirements. The section is intended to demonstrate that, while the incorporation of performance is important and can occur throughout the SDLC, it's best implemented with planning and foresight and in response to specific needs and requirements.

As the number of stakeholders for a software project increases, the importance of formalized requirements commensurately increases. Where stakeholders represent developers who may be working together to build separate components of software, requirements help ensure they are developing toward identical specifications and a common goal. The extent to which requirements can unambiguously describe the body of development work enables developers to rely on information rather than interpretation. Stakeholders rarely enter project planning with identical conceptualizations of intended software function and performance; however, through codification and acceptance of requirements, stakeholders can inaugurate a software project with a shared view of software objectives and future functionality and performance.

Without collective software requirements, at software completion, stakeholders may disagree about whether software is high quality, low quality, or even complete. Without an accepted body of requirements, stakeholders have nothing to assess software against except their own needs, which may vary substantially by individual. Moreover, undocumented needs have a tendency to morph throughout the SDLC through scope creep. Software requirements help avoid the misalignment that occurs when the software produced either exceeds or fails to meet customer needs, discussed in the following sections and demonstrated in Figure 2.4.

Note that while gold-plating may provide additional functionality or performance, this does not result in increased software quality because the customer did not need, had not required, or would not gain additional business value from this additional effort.

Measuring Quality

Quality is a respected pursuit but even ambition cannot overcome an aimless journey toward an unknown or undefined destination. Backpacking through Central and South America, I rarely knew which chicken bus or taxi I was taking or where I would be sleeping, but I always knew the end-game: a flight out of Buenos Aires months later. Because the flight was defined, it was a clear metric that I would achieve, miss, or possibly modify to extend the journey. While backpacking, I could continually alter my route, travel modalities, and sightseeing to ensure I arrived in Buenos Aires on time.

The quest for software quality can be a similarly convoluted and tortuous pursuit but it should have a defined destination. Through performance requirements, SAS practitioners are made aware of the necessary quality characteristics that must be built into software. Even within Agile development environments that implement iterative, rapid development, the expected functionality and performance are defined and stable within each time-boxed iteration. Performance requirements not only guide and spur design and development, but also should be used to evaluate the attainment of quality at software completion and during operation.

Throughout the dynamic performance requirement chapters, the Measuring sections demonstrate ways that performance can be quantitatively and qualitatively measured. Dynamic performance requirements more readily lend themselves to quantitative measurement than static performance requirements. You can specify that software should complete in 15 minutes, be able to process 3 million observations per hour, only fail less than two times per month, or be operational within Windows but not UNIX. Dynamic performance requirements are not only measurable but also observable; thus, they are typically prioritized over static performance attributes in requirements documentation.

Static performance requirements are often more appropriately measured qualitatively or indirectly. For example, modularity might be approximated by counting lines per macro, or ensuring that all child processes generate return codes to signal success or failure to parent processes. However, modularity is more likely to be expressed as a desired trait of software and thus measured qualitatively. Some development environments similarly track how many times specific modules of code are reused and in what capacity or software, which can provide reuse metrics from which reusability can be inferred. But the best measurement of the inclusion (or exclusion) of static performance requirements is an astute SAS practitioner who can inspect the code and discern quality characteristics.