15.4.1 Evolution of the Approach Over Time

The report initially gained credibility within R&D, but has now spread far outside R&D to the highest executives. This has affected how each year’s theme is chosen and how follow-up is done. Twelve years ago the theme was chosen by Avaya Research, but for the last several years it has been chosen by corporate executives. The report evolves primarily based on the changing goals of the corporation.

Initially, we conducted follow-up sessions on the report with R&D management. Now we also meet with personnel from product management, quality, and services, as well as with Avaya business leaders. A version of the report is tailored for each organization based on data that focuses on that organization and recommendations that are the most relevant for it. For each recommendation we now suggest a role responsible for implementing the recommendation. Because of resource constraints in development we focus on the top few recommendations appropriate for an organization in order to increase the probability that action will be taken by that organization.

When Avaya was formed, there was a heavy emphasis on time-to-market. Since Avaya products can be used in mission critical situations corporate goals more recently place a heavier emphasis on product and solution quality [10]. There has also been increased emphasis on making R&D more efficient. This has affected what data are gathered and what analysis is performed.

The number of acquisitions has increased over time, bringing new issues, cultures, practices, market areas, and repositories. This has affected both the data gathered and the analyses performed.

Avaya products are increasingly interdependent, and the company is transitioning from a product management model to a “solution” development model. Along with this comes new data, such as interoperability matrices, that we use in our analysis.

The next two sections contrast what has changed with what has remained the same.

15.5.1 Data Accuracy

In addition to new data migration challenges noted previously, assuring and estimating the accuracy of our data is a major concern. We deal with large datasets that are distributed across a variety of repositories, and for which some of the data are entered manually, often by people far removed from the analysis of the data. For example, MR data are stored in different configuration control repositories and the descriptions of the changes in the MRs are entered by software developers or by support teams that may not be aware that their entries become part of a company-wide analysis. Human error or even just vagueness in such descriptions can be a major source of inaccuracy, as discussed and shown in Ref. [4]. The problem is compounded when performing analyses that include several different sources or types of data. We believe it is incumbent on us to estimate the error in the analyses that we do, and we continue to seek good methods for doing so. This is a research topic that the measurement community should address. In the following sections, where we present some examples of our analyses, we will provide error estimates where we can reasonably make them.

15.5.2 Types of Data Analyzed

As previously noted, our quantitative data sources tend to focus on the process by which the requirements are obtained and tracked, the process by which software is developed, the demographics surrounding development, and the quality as seen by the developers and by the customers. Each area has a rich set of tools that can be used to quantify the history of events in the area and how they relate to, for example, customer-perceived quality downstream. Code commits and MRs are used to extract information about a variety of issues, including types of errors that occur and practices that are used. For example, by examining when an error is found we can usually determine the detection technique used, such as a code review or a system test. It is difficult to spot corporate-wide trends by looking at code measures since languages, platforms, and development tools vary across the company; we leave such analysis to the individual projects. However, as projects have deployed various static analysis tools, such as FindBugs and Coverity, and inspection tools such as Atlassian’s Crucible, richer sources of data have become available.

15.6.1 Demographic Analyses

Questions: How is staff distributed across locations and divisions?

How much staff churn is there?

The reports look at the distribution of the current R&D population by location, titles, experience, division, company of origin, and by type of staff (employee, contractor, outsource, or offshore). Understanding the distribution and composition of the R&D organizations helps us to understand problems R&D is facing and to spot trends. For example, Figure 15.5 shows that many divisions in Avaya were spread across many locations, prompting us to study issues that projects have with distributed development. Figure 15.6 shows a major buildup in the use of contractors and in outsourcing, and so we are drawn to address issues such as knowledge transfer and how work is partitioned.

Both figures are based on an analysis of Avaya’s corporate personnel directory, which includes information on the role and location of every employee. In addition, some data on outsourced teams is obtained by interviewing Avaya staff that is tasked with managing outsourced teams.

Analyzing the origins of the company’s current employees provides insight into products, technologies, and markets with which they are familiar. This has impact on software development methodologies and on the locations of experts in various disciplines. A more detailed view (not shown here) reveals for each year hired how many employees remain and how many of the original set have left the company. This can be used to examine retention policies and morale issues.⁵

The demographic trends can be combined with other trends, such as code growth, to show where support may need to be rebalanced.

Another analysis displays staff churn trends, that is, the turnover rate at different locations, which allows us to estimate the rate of loss (or gain) of domain expertise. Because real-time telecommunications products are quite complex, it can take 6–12 months or more before a developer new to the product or company can be effective. In more complex projects we observed that it can take 3 years for a developer to become effective [13]. The churn trend can be used in analyzing productivity losses caused by staff loss or by training rates for new staff. Figure 15.7 shows long-term trends in Avaya product experience, and, along with more detailed charts, such as Figure 15.8, it is used to recommend what an appropriate balance of new to experienced staff should be based on historical data. The acquisitions demonstrate where infusion of developers new to Avaya occurs. Figure 15.8 shows a comparison of the geographic distribution for two specific points in time. Variations of this chart for individual organizations or functions highlight where the experience is out of balance and corrective action is needed. We do not have experience history from all acquired companies so the experience in the worst case could be understated by as much as 11%, but is more likely under 5%.

Our demographic analyses are based on our corporate personnel directory. We create a snapshot of the personnel directory every month and use this data to analyze trends.

In summary, the demographic analyses provide a background and context that assist us in understanding other significant trends in the company.

15.6.2 Analysis of Predictability

Question: How well do Avaya projects predict their completion dates?

Avaya employs a business process with gated reviews for all projects to move projects forward and to synchronize such functions as development, training, documentation, services, installation, and customization. Depending on project characteristics, Avaya uses a variety of processes within the development phase, including iterative and traditional waterfall processes. Predictability of development is important to make sure all functional areas allocate resources appropriately, including areas such as design, system test, interoperability test, documentation, globalization, localization, services for alpha trials, beta trials, general introduction, and support. Data are provided to and tracked by per-product Product Management Teams.

We are able to extract planned and actual dates from a number of sources and compare them for consistency. Figure 15.9 shows how well projects predict their completion dates at different gates in the development cycle. The y-axis shows how late or early a project completion estimate is relative to its actual completion. The horizontal line indicates on-time completion. The distribution of estimates at different gates in the development cycle is shown as shaded ovals. For example, at the OK-to-Plan gate, which marks the end of the feasibility phase, the shaded oval shows the distribution of estimates. The median estimate is indicated by a small black horizontal oval within the larger oval. One can see that very few projects fulfill their estimates. Furthermore, even when development has advanced to a later gate, there is only a small improvement.

Figure 15.9 was created by analyzing the predicted and actual project business gate dates for all Avaya projects over an 18-month period in 2004 and 2005. Schedule prediction is now closely tracked by Avaya’s operations team and schedule performance has substantially improved

Figure 15.10 shows a relation between three factors and projects’ ability to predict their schedule. For example, one factor is the degree of cross-project and cross-division cooperation required. We found that projects involving cooperation among divisions were worse at predicting completion dates than projects that were located within a single division but had development staff at several different sites, which were worse than projects that were located at a single site. Without quantitative data an experienced project manager might guess the same result, but would not know the magnitude of the difference. We also look at predictability by phase and predictability by variables, such as project complexity or size, number of sites, or development methodology. We track the predictability trend to ensure that practices are being put in place to improve predictability. At the time the charts in Figures 15.9 and 15.10 were generated we observed that Avaya was not good at prediction and that Avaya’s ability did not improve as the development cycle progressed. Focusing attention on this issue through quantitative analysis led to the use of better prediction techniques and close tracking by Avaya’s operations teams. Subsequent analysis showed improvements. This is a good example of an improvement resulting from (quantitatively) highlighting a problem.

Figure 15.10 shows factors that appear to be correlated with predictability. It was created by analyzing the predicted and actual project business gate dates for all Avaya projects over an 18-month period in 2004 and 2005. We identified project size and organizational complexity from project staffing profiles, and determined the product complexity from requirements and design documentation.

Our primary data sources for predictability analyses are the committed and actual dates that are tracked in Avaya’s program management repository. These data are updated and tracked on a monthly basis and we estimate the data is about 95% accurate because it is carefully reviewed by Avaya’s program managers.

15.6.3 Risky File Management

We observed, based on empirical analysis, that 1% of project source code files contain changes for more than 60% of the customer reported defects in most Avaya products. The ability to identify these “1%” files helps prioritize risk mitigation activities.

Risky File Management (RFM) [14] helps to prioritize and determine the most appropriate risk mitigation actions. We have refined this approach while working with over 50 Avaya projects. Briefly, it involves annotating the source code at the file and module level with the historic information needed to identify risk, and proposes the most suitable actions to mitigate the particular types of risk. It is somewhat similar to approaches used to produce highly reliable software for critical situations, such as described in Ref. [15]. However, in contrast to work described in Ref. [15], the purpose of risky file management is to prioritize risk remediation by selecting risky areas and the remediation approaches that are likely to be most cost-effective. For example, to advocate static analysis techniques described in Ref. [15], we tried to find (but did not) instances of a defect flagged by static analysis techniques that caused a failure reported by a customer. In contrast, several projects reported a non-negligible fraction of fixes to static analysis warnings that introduced new defects. As a result, we prioritize fixes only for certain classes of static analysis defects and only for selected areas of the code.

The data we use comes from version control and MR tracking systems. Software projects use a version control system (VCS) to keep track of versions of the source code. Each revision of a source code file a developer makes is “committed” to the VCS. Information about the commit that can be retrieved includes author, date, affected file and its content, and a commit message that typically contains related Modification Request IDs. Examples of VCS include Subversion, ClearCase, Git, Mercurial, CVS, Bazaar, and SCCS. Developers create branches of a project source code to work on a particular release. VCS typically support creation of a branch so that changes made to one branch can be tracked separately from changes made to another branch. Most projects have at least two branches: a development branch with the latest features, and the release branch (for code that is in the product released to customers), which is unchanged except for important bug fixes. MR/Issue tracking systems are used to track the resolution of MRs (each commit is typically associated with an MR). The MR can be thought of as a software development task. The task may be to implement a new feature or to fix a defect. Among attributes associated with an MR is one that can be used to determine if the MR was customer-reported, so we can tell which defects were discovered by customers. Common issue tracking systems include Bugzilla, JIRA, ClearQuest, and Sablime.

The following lists the main steps of the RFM approach:

1. Data collection and analysis
We gather data about each version of every file in a set of source code repositories for the majority of Avaya projects, and then prioritize the files and identify a candidate set of riskiest files (about 1% of all files) using a weighted algorithm based on the empirical results. While we create the risk profiles based on all files in all Avaya source code repositories, each project is presented with the most risky files that are in repositories related to that project. We obtain the following information for each revision of every file in each source code repository:

1.1. Path name, which uniquely identifies the file in the repository. Pathnames will be different for each branch in some of the version control systems, for example, in SVN.

1.2. Author, date, commit message, and content. We process the content of the file to obtain an Abstract syntax tree (AST), and size (in lines of code). We also process the commit message to identify MR identifiers, if present.

1.3. To determine equivalence classes for a file, all versions of all files obtained in step 1.2 are examined as follows:

1.3.1. If some version v1 of file f1 matches some version v2 of file f2 (matches means that they have identical content or that they have an identical abstract syntax tree (AST)) the files f1 and f2 are considered to be “related”. Typically, the same file may be modified in multiple branches or even different repositories (when the same code is used in multiple projects).

1.3.2. The “related” relationship is transitively closed, that is, if f1 is related to f2 and f2 is related to f3, then we declare that f1 is related to f3 even if f1 and f3 may not have a single version with the same content or AST.

1.3.3. Open source software (OSS) files are identified by matching each version of each file for each equivalence class of the related files identified in step 1.3.2 to the large repository of open source code with more than 200M unique versions mentioned earlier. If a match is found, then the equivalence class is considered to be an OSS file.

1.4. The corporate personnel directory is accessed to determine for each author obtained in step 1.2 if the author is an active employee. If the author is an active employee the employee’s name, phone number and e-mail address are obtained. Any other information available in the corporate directory could also be obtained.

1.5. We identify customer found defects (CFDs) and other MR attributes by matching MR identifiers obtained in step 1.2 to data in the MR tracking system.

1.6. For each equivalence class of related files, aggregate the data over all related files to obtain the number of commits, number of authors, number of authors who left Avaya, and the number of CFDs.

1.7. Obtain an empirical relationship between properties of the file and CFDs using statistical models using all commits to the project’s version control system for a period of time (typically a 3-year period) and fitting a logistic regression model with an observation representing an equivalence class of related files, with the response being whether or not any of the files in the equivalence class had a CFD, and the predictors including those described in step 1.6. For example, in several projects the most important predictors of future CFDs are the number of past changes, the number of SV MRs, and the number of authors who left Avaya.
The fitted model coefficients are then used to prioritize the list of candidate riskiest files.

2. Presentation of the analysis results to relevant stakeholders.
The approach provides a subject matter expert exploration view, an online dynamic table, and a downloadable spreadsheet. This is the critical and important part of the approach, as it provides information tailored to the decision-making needs of different stakeholders. Figure 15.11 shows an example of the subject matter expert exploratory view. The candidate-risky files and two most recent CFDs are identified in the first column. The second column contains the number of MRs requiring changes to the file or a related file and a hyperlink to a list and description of each MR. The third column contains the number of authors who have changed the file or a related file and a hyperlink to the list of authors. The third column also contains the number and percentage of authors who are no longer in the company. The fourth column contains the number of related files and a hyperlink to the list of related files

Results from different tables or different portions of the table can be tailored to the needs, interests, and skills of a particular stakeholder. The following are a few examples:

• A project manager sees the priority of the risk and an estimate of resources or time to remediate the risk.

• A subject matter expert sees more technical details of what the type of risk is, what underlying technical details led to the file or module being classified as risky, and so forth.

• A development manager sees who made changes to the file and can use that information to identify potential owners of the code or to identify reviewers for design or code reviews.

Table 15.2 shows key data needed by subject matter experts to reach the most appropriate risk remediation action.

3. Remediation actions. A checklist of heuristics based on experience and empirical data to help the expert take the most appropriate action: for example, no action, control program, or reengineering. For each candidate or indicated risky file, the subject matter expert can analyze the file or module and any associated data. The system can provide an optional guideline, based on empirical data of previous actions taken to remediate risks.

3.1. No action may be required, if, for example, development is complete for this file; the candidate file will not be used in the near future; the candidate file is changed with a risky file, but is not itself risky.

3.2. The subject matter expert can recommend a control program involving additional review and testing of all changes to the file, or creating additional documentation to make clearer the design and implementation issues considered in producing the file. Such a control program mitigates risk from changes to the file. For example, if the file has many authors or other reasons warrant, the file owner can create a 1-page design guidance document that is available for anyone who changes the file. The same design guidance document may also apply to a set of files that all contribute to the same component or feature.

3.3. Finally, the subject matter expert can recommend that the file be reengineered if development is active and the file is deemed to be fragile and difficult to change without introducing more risk.

Typically, the subject matter expert works with the development manager and project manager to schedule any recommended changes to the file. Figure 15.12 shows the distribution of actions taken in response to identification of risky files by an example project.

15.7.1 Original Seven Key Software Areas

The following seven practice areas were analyzed until 2011:

• Customer-focused development: Practices that emphasize customer input and feedback throughout the development lifecycle, for example, Root Cause Analysis, Customer Feedback, Front End Planning, Increase Customer Understanding, Empower Product Owner.

• Design quality in: Practices that provide a focus on quality early in the development lifecycle, for example, Adherence to Internal Technology Standards, Architecture, Baseline requirements, Build Management, Code Inspections, Collaborative Product Team, Cross Division Architecture reviews, Design Reviews, Interface Specifications, Management of third Party Deliverables, Refactoring, Reuse.

• Improve testing practices: Practices that improve the automation and comprehensiveness of developer and system tests, for example, Code Coverage, Memory Leak Detection, System Test Automation, Stress and Load Testing, Unit Test Automation, Test Focused Development.

• Software project management: Practices to support planning, monitoring and controlling an individual software project, for example, Cross-Division Cooperation, Knowledge Transfer, Measurement, Predictability, Product Management and R&D Learning, R&D Skills, Release Tracking.

• Multisite/offshore development: Practices that support the effective development of software across geographic boundaries, especially those involving offshore teams, for example, Cultural Training, Decouple Work, Ease of Communication, Empowered Teams, Knowledge Transfer, Multi-Site Development Environment, Trust.

• Architecture-guided iterative development: Practices drawn from agile and traditional methodologies that are organized into a family of iterative processes, where development is guided by a well-defined architecture, for example, Agile: Collaborative Product Team, Customer Feedback, Daily Stand-up, Document Just Enough, Ease of Communication, Empower Product Owner, Empowered Team Lead, Iteration Retrospective, Prioritized Feature List, Refactor, Test Automation, Test Focused Design, Time Boxed Iterations, Track Iteration; Traditional Architecture, Baseline Requirements, Build Management, Code Inspections, Manage third Party Deliverables, Track Releases.

• Cross-project cooperation and coordination: Practices that support cooperation and coordination across individual project boundaries. Such cooperation is required for platform-based development and solution-based development.

Our assessment for each practice was based on:

• Criteria for effective deployment of the practice based on accepted industry practices, such as those published in the International Conference on Software Engineering or IEEE Software, and Avaya good practices.

• The extent of usage based on input sessions and our partnerships with Avaya projects.

• Self-assessment of the effectiveness of the practices by a project and the reasons for that assessment.

Each practice was plotted on a grid showing effectiveness and extent of deployment (see Figure 15.13).

The areas and practices were kept relatively consistent until 2010 so that we could analyze trends and make recommendation based on those trends. In addition, several practices within the practice areas were selected as indicators of projects that focus on quality, as discussed in the next section.

15.7.2 Four Practices Tracked as Representative

Beginning in 2012, the following four practices are tracked by the Avaya quality council as necessary, but not sufficient, for achieving good quality while meeting schedules⁶ :

• static analysis, using a commercial or open source tool;

• code review and inspections;

• automated regression testing, to prevent breakage and provide acceptance criteria for code being delivered to test or other projects;

• code coverage, to identify areas of the code that had not been adequately tested.

The report originally started tracking these as a group in 2011, contributing to the Avaya Quality Council’s efforts to standardize and focus on a small set of representative good practices. Objective targets for these practices were established so that it was clear what actions were expected of projects. The scores of these four practices are averaged for each project to provide a development process measure that is easy to discuss with the quality council and with R&D leaders.

A project’s practices are assessed at each product development business gate, including prior to the start of development, when implementation is complete, for example, prior to beta testing, and prior to product launch. We have found these measures to be predictive of the quality of the end product [16].

15.7.3 Example Practice Area—Design Quality In

The following 11 individual practices are representative individual practices associated with the “Design Quality In” practice area.

• Baseline requirements and place them under change control.

• Specify and update an architecture sufficient to guide development.

• Deploy internal technical standards.

• Create well-defined interface specifications [17, 18].

• Perform cross-division architecture reviews [19].

• Establish a component integration and reuse program.

• Review requirements, interface specifications, design artifacts, and test scripts.

• Perform inspections of new or changed code.

• Perform automated build management (at least daily) with automated sanity tests [12].

• Carefully manage third party deliverables and dependencies on other projects.

• Refactor a criteria-based selection of modules.

Each of these practices may be further partitioned into sub practices. For example, in Section 15.8.1.1, we identify six key sub practices of the automated build management practice.

We have defined criteria for effective deployment of each of these practices. For example, the criteria for effective deployment of “manage 3^rd party deliverables” are the following.

• All third party deliverables are identified.

• Quality policies are in place and communicated to third party owners.

• Policies are in place on conditions for accepting updates.

• An acceptance test program is in place for each third party deliverable.

• A development team member is identified as “local owner” of each third party deliverable.

• Schedule and quality impact of third party deliverables is assessed.

An assessment of the individual practices for “Design Quality In” from several years ago is shown in Figure 15.13. The x-axis represents how widely the activity is deployed (Usage), and the y-axis represents how effectively the process has been deployed (Effectiveness) based on criteria for each practice. The evaluation of effectiveness and usage for a practice is a judgment based on our discussions with R&D project members and our analysis of quantitative data. In Figure 15.13, the judgments are represented by the positions of the black circles. For example, at the time of the assessment automated build management was widely deployed and Avaya R&D projects were highly effective in performing automated build management. On the other hand, there was limited usage of structured refactoring techniques [20] and where deployed the practice was judged to have medium effectiveness. The Overall circle, in blue (dark gray in print versions), represents our judgment of the usage and effectiveness of the total set of “Design Quality In” practices, which was medium to high usage and medium effectiveness.

These charts provide guidance by identifying software practices to target for improvement. Avaya provides guidance on best practices to projects, but leaves the implementation of each specific practice to each project.

Figure 15.14 shows our overall assessment of design quality in from 2002 to 2008. This practice area shows initial improvement, then decline followed by slow improvement in usage and minimal improvement in effectiveness. Avaya has inherited a long tradition from Bell Labs of quality products and quality-focused development processes (see, e.g., [21]), and quality remains a strong emphasis for Avaya R&D leaders and staff.

15.7.4 Example Individual Practice—Static Analysis

When a few products were not meeting quality goals many business units within Avaya made the use of static analysis mandatory. Projects had a choice of tools—either a centrally funded commercial tool or open source tools such as FindBugs for JAVA. Quality councils tracked how frequently static analysis was used and how aggressively projects fixed real violations. This proactive effort to remove defects complemented code inspections and automated testing.

We analyzed the strategies that projects were using to remove violations, tracked trends across projects that had used static analysis for multiple releases,⁷ and summarized good project policies in addressing static analysis defects.

For example, depending on product quality, project stage, staff expertise and other characteristics, one or more of the following static analysis policies are typically used by Avaya development teams:

• no new or high impact violations allowed in any build,

• target critical violations (potentially high impact outliers),

• decrease or eradicate medium impact violations,

• cover fixed code by automated tests,

• fix when a file is being changed for other reasons, especially for legacy systems.

When fixing violations in legacy code where there was limited or no experience remaining with the file, projects took two approaches to minimize breakage:

1. If a file was being changed, and the developer had some familiarity with the file, the project took the opportunity to correct high and medium impacting violations.

2. A project addressed all violations of a particular type in legacy code all at once, regardless of whether a file was being changed for some other reason.

We found that a few types of violations accounted for the majority of violations. We also found that some violations had a more critical impact on the code than might be suggested by the vendor. We gave guidance on what type of violations to fix first. Figure 15.15 shows the distribution of the top violations across a sample of 32 Avaya projects. For these 32 projects, over 26% of all static analysis violations are “pass by value” or “uninitialized constructor” violations.

Similar charts are created for individual projects.

We track to determine if product teams are improving in their deployment of static analysis practices. Figure 15.16 shows an example for one Avaya R&D organization. The average static analysis score for the organization’s product teams as determined by the R&D quality council increased year-over-year for most years leading up to 2013. In addition, the number of product teams with a static analysis score of 4⁸ (the largest score possible) increased each year.

15.8.1 Example Recommendations

Recommendations made between 2002 and 2013 cover a variety of topics, such as the following:

• front end planning;

• deployment of architecture-guided iterative and agile practices;

• software project management;

• multi-site and offshore development practices;

• domain expertise of the R&D staff.

The following sections describe two example recommendations. The automated build recommendation is an early recommendation (Section 15.8.1.1), initially made in 2003. The risky file management recommendation (Section 15.8.1.2) is more recent, initially made in 2012.

15.8.2 Deployment of Recommendations

Recommendations are deployed in partnership with business and R&D leaders. We conduct follow-up sessions with individual business leaders, R&D leaders, and product management leaders of each organization. The sessions are typically structured based on the following list of questions for each recommendation.

• Is the recommendation relevant to your organization/team?

• If relevant, is it already deployed in your organization/team?

• If relevant and not deployed, what are the barriers to deployment?

• What actions are you prepared to take?

• How can our team help in deployment?

Note that as a result of these sessions, a recommendation may not be deployed in a particular organization either because it is not relevant to that organization or the barriers for deployment are too large. In addition, the recommendation may be adapted to make it more deployable in a particular organization. When appropriate actions for an organization are identified for a recommendation, action plans are then defined, deployed and tracked. Results are incorporated into the next year’s assessment report.

15.9.1 Example: Automated Build Management

Our assessment in 2002 and 2003 was that Avaya projects had medium to high usage of automated build management practices with low to medium effectiveness¹⁰ as shown in Figure 15.17. In 2003 and 2004, we made specific improvement recommendations as described in Section 15.8. These recommendations helped create a focus in Avaya R&D in 2004 and 2005 to improve build management practices. The good practices in build management described in Section 15.8.1.1 are now widely deployed and build management is an effective, efficient practice for most Avaya projects (see Figure 15.17) for 2002 through 2008.

Automated build management continues to be a strength of Avaya product teams in 2014 with widespread and effective deployment. In fact, our assessment focus is no longer on automated build management, but on continuous build and integration practices.

15.9.2 Example: Deployment of Risky File Management

We provided briefings on risky file management to about 40% of the active projects in the first half of 2013 (i.e., the first several months after the 2012 State of Avaya Software report was published).

• 50% of those projects’ products deployed risky file management as a part of their development process.

• 25% of the projects are considering deployment.

• 25% of the projects are not deploying, typically because the project is no longer doing significant new development—that is, they are now doing sustaining engineering.

We do not yet have post-release quality data for most of the projects that deployed risky file management, because they have not yet been deployed long enough to obtain reliable defect data. Two projects with sufficient defect data that used risky file management as part of their quality improvement program had significant improvements in quality.

• The customer quality metric (CQM) [1] for an Avaya contact center product improved by 30%.

• The CQM for an Avaya endpoints product improved by well over 50%.

15.9.3 Improvement in Customer Quality Metric (CQM)

The operational cost of fixing a customer found defect includes the cost to diagnose the defect, to implement and test the fix to the defect, and to deploy the corrected fix.

We estimate the impact of the assessments based on the operational savings as a result of fewer customer found defects (CFDs).

The operational benefit for any product release is calculated as follows. First, we calculate or obtain the following data.

• Expected number of CFDs against the product release; the expected number is based on CQM (which is based on in-service time per customer) prior to improvement and the number of product release installations.

• Actual number of CFDs against the product release.

• Operational cost of fixing a CFD (cost provided by Avaya corporate quality team).

Then,

Operational benefit = Operational cost of fixing a CFD * (expected number of CFDs – Actual number of CFDs)

Table 15.4 shows the operational savings calculations for a sample of product releases over the past 3 years.

The prior release of Product C (Release 6.1) did not deploy practices recommended in the reports and had numerous quality issues resulting in many millions of dollars in additional operational costs.

Using the approach described in this section, we estimate the annual operational savings based on the deployment of quality focused improvements for the past 3 years to be at least $60M per year based only on reduced cost of dealing with customer reported issues, and not including benefits resulting from increased customer satisfaction. The investment in the quality improvement effort was likely to be much lower. Approximately 20% of the effort of the ARC team goes into producing the report and obtaining relevant measures. The investments made by development, testing, and quality organizations are harder to quantify, but are highly likely not to have exceeded the savings.

As noted in earlier sections, the process of assessing software development and sustainment in Avaya has had significant impact, on the way that software is developed, on the practices and processes targeted for improvement, on achieving company goals, and on the cost and value of software technology within the company.

The impact has been made possible by focusing on the goals of the company and on obtaining data that can be used to analyze how well software development within the company is contributing to meeting those goals, following the GQM approach to empirical studies. The idea has been to improve the way the company does software development and to know it.

The impact of the process is evident in several ways. The annual analysis, as described in the report The State of Software in Avaya, shows to all who are interested what improvements have occurred and what areas need improvement, often prompting allocation of resources to making improvements and monitoring those areas. It helps the company set goals and understand better whether or not it is achieving those goals. For example, improving software quality by reducing defects found by customers frees software developers’ time to work on adding new features and capabilities to products rather than correcting bugs in those products. With appropriate focus on data collection, we are able to estimate how much the savings from improved quality is, providing more substance to the idea of knowing what the improvement is.

The assessment process can also be viewed as providing a competitive advantage to the company. It both allows the company to show customers the quality improvements and allows more rapid development of new features or customization to customers’ needs. As a result, the company both retains existing customers and attracts new customers, instilling confidence in customers that the company has a systematic process in place to improve quality.

15.10.2 Factors Contributing to Success

The assessment process was started in 2002 and is a continuing process, producing the State of Software in Avaya report at the end of every year. Based on its longevity and continuing impact, we believe the assessment process has been a success. Factors contributing to that success include the following:

1. a well-defined measurement methodology, founded on the goal-question-metric approach [3, 4, 24];

2. the ability to identify organizational goals;

3. the availability of the necessary data and the willingness of the producers of the data to share it;

4. cooperation by people at all levels of technical and management positions, from software developers to the company’s executives;

5. confidence by all involved that the conclusions drawn from analysis of the data, both quantitative and qualitative, are valid, with such confidence instilled by continuing discussion with development organizations, and retrospective views of the results;

6. acknowledgement from the software development organizations that the annual recommendations are useful and lead to (needed) improvements;

7. evidence that the assessment process is having a positive financial impact on the company, for example, that considerable time and effort is saved through improvements in quality of the type and by the means suggested in the state of software reports;

8. an incremental approach to implementing the assessment program, starting with relatively modest goals and an in-house focus, using a subset of the types of data that are currently analyzed, and proceeding over the years to a broader focus incorporating customer-focused data into the analysis.

In summary, the assessment process started with clear goals, a repeatable, systematic measurement process, gained the confidence of those who provided the data by providing them with useful analyses and recommendations, and then expanded the focus of the analysis and the breadth of the data analyzed. During the 12 years, so far, of the assessment process, the company has undergone significant changes in management and personnel, but the utility of the assessment process has remained clear to all, and it has continued to receive strong support from the software development organizations, from company executives, and from Avaya Research.

15.10.3 Organizational Attributes

The success of the process, of course, is also greatly owed to the skills and commitment of the ARC who, from the beginning, formed close relationships with the development organizations who both provided the data and were the subjects of the recommendations. The ARC was (and is) principally staffed by two or three people with considerable development experience and experience in working with development organizations to foster improvements. The ARC was established by a research director and was a part of a research department, and received direction from the director, and not from someone in the direct management line of the development organizations. Accordingly, it was perceived as being objective and non-competitive. It was also able to call on others in the research department for support in discussing and implementing different forms of data analysis. Furthermore, the ARC has remained a stable organization, with two of the same members that it had at its inception, and with continuing support from the same researchers.

15.10.4 Selling the Assessment Process

As previously noted, there are a number of factors contributing to the success of the assessment process and the corresponding annual report. Perhaps most important to the success of the process was showing that the findings of the reports could have significant impact on the company by improving software development and related practices, and by making stakeholders out of a wide variety of people in roles throughout the company, ranging from typical software developers through the company CEO. Without continuing support from the stakeholders, the assessment process and report would have had no way to demonstrate value. Early in the process, we worked with small groups who had confidence in the ARC based on earlier dealings with its members. Based on the early results, as the process continued we broadened the stakeholder base, and always made sure we were focusing on issues of interest to stakeholders, often by showing them preliminary results and asking for their validation. As the process evolved, we could show the results of the recommendations from earlier analyses, thereby helping to validate the value of the process, and establishing a virtuous cycle. We think that such an approach helps assure success and would advise others who are interested in starting an assessment program to use a similar strategy.

15.10.5 Next Steps

We will continue to prepare an annual state of software report for Avaya and expect to focus on the following areas in the coming years.

• Our demographic assessments focus on the R&D development and testing community and, because of the role of software across Avaya’s business, we intend to include product management, services and support as well as other areas in future demographic assessments.

• As noted in Section 15.6, we have discovered that it is even more important to help projects implement recommendations than to provide them. As a result, we will continue to provide integrated analyses of multiple data sources and tools and procedures to aid in the action steps for individual projects. Risky file management (Section 15.6.3) is an example of this approach.

• As we utilize new data repositories (Section 15.5), we will improve our ability to estimate the accuracy of our data and analyses.

We are also interested in benchmarking with other companies who are performing similar analyses and with whom we could assure data comparability.

It is an open question as to how the various factors analyzed interact with each other, and is a subject of future research.

The techniques we use in producing the report could be applied in any organization that can define its goals, that is willing to collect the data needed to measure progress towards those goals, and that is willing to make an investment in improving its development practices. Although we typically think of such organizations as companies, or business units within companies, the techniques could be applied to any entity that develops software. For example, a worthwhile goal might be to produce a report on the state of software in the USA. Certainly, the challenges would be large, but the pay-offs in terms of knowing where one should invest to improve the way software is developed in the country would be large as well.

As one source of input, we use individual and small group input sessions with R&D and product management. These sessions cover all divisions, major R&D sites, various levels of technical staff and management, and a variety of R&D roles, including architects, developers, project and program managers, testers, documentation, and support. Our findings are reviewed with them and adapted based on their feedback. Each session is based on two sets of questions and a set of software practices that we tailor based on the role of those in the input session. Common questions include:

• What are the three primary software-related areas that your project is doing particularly well?

• What are the three primary software-related issues facing your project?

• What is your assessment of the software quality for your project and why?

• What is your assessment of the productivity for your project and why?

• What is your assessment of the domain expertise of your staff? Why?

• What software development approach(es) are used in your project?

• If you could change one software related area in Avaya, what would you change?

The nature of these sessions and the questions that we ask has evolved over time. For early reports we had less data on trends and on a variety of factors, such as productivity, cost, and quality. Accordingly, we spent more time in individual discussion sessions. Currently, we are able to show session participants charts and graphs showing trends and, as necessary, data about their organizations, and use those to focus the session. As a result, sessions are more to the point and briefer.