Key concepts you will need to understand:
✓ System-development methodologies and tools (prototyping, RAD, SDLC, object-oriented design techniques)
✓ Documentation and charting methods
✓ Application implementation practices (piloting, parallel run)
✓ Software quality-assurance methods
✓ Application architecture (client/server applications, object-oriented design, data warehousing, web-based applications, interfaces)
✓ Testing principles, methods, and practices
✓ Project-management principles, methods, and practices (PERT, CPM, estimation techniques)
✓ Application system-acquisition processes (evaluation of vendors, preparation of contracts, vendor management, escrow)
✓ Application-maintenance principles (versioning, packaging, change request)
✓ System migration and data-conversion tools, techniques, and procedures
✓ Application change-control and emergency change-management procedures
✓ Post-implementation review techniques
Techniques you will need to master:
✓ Evaluate the processes by which application systems are developed and implemented to ensure that they contribute to the organization’s business objectives
✓ Evaluate the processes by which application systems are acquired and implemented to ensure that they contribute to the organization’s business objectives
✓ Evaluate the processes by which application systems are maintained to ensure the continued support of the organization’s business objectives
Organizations invest heavily in the development, acquisition, implementation, and maintenance of applications and their associated systems. These applications are developed or acquired in support of key business functions. The IT department should have clearly defined processes to control the resources associated with the development, acquisition, and implementation of applications. This process, called the systems-development life cycle (SDLC), encompasses a structured approach to do the following:
Minimize risk and maximize return on investment
Reduce software business risk, the likelihood that the new system will not meet the application user’s business expectations
The lack of a formal documented software-development process can result in software projects that are not on time or on budget, or do not meet user or business needs. The IS auditor should look for evidence of a structured approach to application development, acquisition, implementation, and maintenance. The IS auditor also should review policies and procedures to ensure that the objectives of the strategic plan are being met. The SDLC should have clearly defined life-cycle phases and progression points, to allow the IS auditor to identify each phase in the process and ensure adherence to planned objectives and company procedures.
As discussed in Chapter 1, “The Information Systems (IS) Audit Process,” all significant IT projects should have a project sponsor and project steering committee. The project sponsor is ultimately responsible for providing requirement specifications to the software-development team. The project steering committee is responsible for the overall direction, costs, and time-tables for systems-development projects.
The IS auditor is responsible for advising the project-management team and senior management if processes are disorderly (informal) or lack sufficient controls. A primary high-level goal for an auditor who is reviewing a systems-development project is to ensure that business objectives are achieved. This objective guides all other systems-development objectives. In addition to auditing projects, the IS auditor should be included within a systems-development project in an advisory capacity to ensure that adequate controls are incorporated into the system during development and to ensure that adequate and complete documentation exists for all projects.
A Software Development Life Cycle is a logical process that systems analysts and systems developers use to develop software (applications). The SDLC should produce high-quality software that meets or exceeds business and user requirements, is produced within time and cost estimates, is efficient and effective, and provides for cost-effective maintenance and enhancement. The most common SDLC is the Classic Life Cycle Model, which can be either the Linear Sequential Model or the Waterfall Method. The SDLC is a framework, and specific activities in the framework will vary by organization. Figure 6.1 shows the high-level phases associated with a waterfall SDLC.
The waterfall methodology is the oldest and most commonly used approach. It begins with the feasibility study and progresses through requirements, design, development, implementation, and post-implementation. It is important to remember that, in using this approach, the subsequent step does not begin until all tasks in the previous step are completed. When the process has moved to the next step, it does not go back to the previous step.
The waterfall approach is best used in environments where the organization’s requirements will remain stable and the system architecture is known early in the development process. Per ISACA, Table 6.1 describes the phases associated with the traditional systems-development life cycle approach.
Table 6.1. Phases of the Traditional Systems-Development Life Cycle
In addition to following a structured approach to systems development, the IT organization must have a sound IT management methodology that includes the following:
Project management—. Utilizes knowledge, tools, and techniques to reach the goals of the project
IT organizational policies—. Ensures that organizational goals are fulfilled and business risks are reduced
Steering committee—. Ensures that the IS department closely supports the corporate mission and objectives
Change-management process—. Ensures that there is not uncontrolled entry of new requirements into the development process or existing systems
To improve software life-cycle processes and software-processing capability, the organization can implement the Software Capability Maturity Model (CMM), developed by Carnegie Melon’s Software Engineering Institute. Software process maturity is the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective. The more mature an organization’s software process is, the higher the productivity and quality are of the software products produced. As software process maturity increases, the organization institutionalizes its software process via policies, standards, and organizational structures. This institutionalization entails building an infrastructure and a corporate culture that support the methods, practices, and procedures of the business so that they endure after those who originally defined them have gone. Figure 6.2 shows the CMM maturity levels.
For the organization to ensure the production of high-quality software that meets or exceeds business and user requirements, it must be produced within time and cost estimates, be efficient and effective, and provide for cost-effective maintenance. This can be accomplished only with an efficient SDLC process.
In addition to the life-cycle phases, the organization must utilize formal programming methods, techniques, languages, and library control software.
The utilization of formal coding standards ensures the quality of programming activities and enhances future maintenance capabilities. Program coding standards serve as a control that enables clear communication among members of the programming team and between the programming team and the users. These standards should include methods of source code documentation, methods of data declaration, and naming standards.
In application development, programs are coded into statements that are then translated by the compiler into machine language. The language chosen will vary by the organization and will be based on the existing environment, skills sets, and application requirements. These are the commonly used programming languages and their attributes:
Common Business Orientated Language (COBOL) and C programming Language—. High-level general-purpose languages.
C++ and Java—. Object-oriented languages.
SH (Shell), Perl, JavaScript, VBScript—. Scripting languages; primarily used in web development.
4GL—. Fourth-generation high-level programming languages; are object-oriented but lack the lower-level detail commands.
Throughout the SDLC, it is important to protect the integrity of source code and executables. This integrity is maintained through the use of production source code and production libraries. The library control software provides access control to ensure, as an example, that source code is accessible only in a read-only state. The organization should have procedures in place to ensure proper access levels and segregation of duties. As an example, users and application programmers should not have access to the production source code.
Per ISACA, IS auditors should perform the following functions during the design and development SDLC phases:
Review the system flowcharts for adherence to the general design. Verify that the appropriate approvals were obtained for any changes and that all changes were discussed and approved by appropriate user management.
Review the input, processing, and output controls designed into the system for appropriateness.
Interview the key users of the system to determine their understanding of how the system will operate, and assess their level of input into the design of screen formats and output reports.
Assess the adequacy of audit trails to provide traceability and accountability of system transactions.
Verify the integrity of key calculations and processes.
Verify that the system can identify and process erroneous data correctly.
Review the quality-assurance results of the programs developed during this phase.
Verify that all recommended corrections to programming errors were made and that the recommended audit trails or embedded modules were coded into the appropriate programs.
Prototyping is the process of developing a system through the rapid development and testing of code. This process uses trial and error to reduce the level of risks in developing the system. In general, prototyping reduces the time required to deploy applications through iterative development and testing. The developers create high-level code (mostly 4G languages) based on the design requirements and then provide them to the end users for review and testing. The end users can then see a high-level view of the system (generally screens and reports) and provide input on changes or gaps between the code and requirements. A couple major challenges are associated with prototyping:
The use of prototyping to develop applications systems often results in overly optimistic expectations of project timelines.
Change control becomes much more complicated because changes in design and requirements happen so quickly that they are seldom documented or approved.
Because of the iterative process, end users might define functions or extras that were not originally defined during the requirements phase. If not properly managed through a review and approval process, it can extend the cost and time required for the project.
The IS auditor should be aware of risks associated with prototyping and ensure that the organization has implemented the proper controls to ensure that the project continues to meet the needs of the organization while providing a return on investment. These controls should be found in the project-management process as well as the change-control process. Both processes should have controls for the regular review and approval of changes in requirements, schedule, or cost.
Rapid application development (RAD) is used to develop strategically important systems faster, reduce development costs, and still maintain high quality. The organization should use a prototype that can be updated continually to meet changing user or business requirements. According to ISACA, this is achieved by using a series of proven application-development techniques within a well-defined methodology:
Small, well-trained development teams
Evolutionary prototypes
Integrated power tools that support modeling, prototyping, and component reusability
A central repository
Interactive requirements and design workshops
Rigid limits on development time frames
Although RAD supports analysis, design, development, and implementation, the initial development is based on requirements that are best understood. This reduces the time associated with traditional requirements gathering and documentation. The requirements (specification), development, and implementation are intertwined, allowing for the delivery of the application in phases. Most organizations have built their processes around a waterfall method of development and might not have the specialized skill sets or supporting processes required for RAD development. The IS auditor must ensure that the processes and skill sets support RAD development.
As stated earlier, there are five distinct phases in the SDLC. In the Waterfall Model, each of these phases has specific tasks and outcomes; the next phase does not start until the tasks and outcomes of the previous phase have been completed. The SDLC helps to ensure the software will meet user and business needs. Therefore, failing to adopt a systematic approach to application development such as that illustrated by the SDLC increases the risk that the system will not ultimately meet business and user needs.
The following sections discuss the detailed tasks and outcomes associated with the five phases of the SDLC.
In this phase, the cost savings of a new system are identified and quantified, and the payback schedule for costs incurred in implementing the system is estimated.
This provides the justification for proceeding to Phase 2.
The need that requires resolution is defined and then mapped to the major requirements of the solution.
It is critical to involve all management and the end-user business function in this phase, to make sure the new system supports business needs:
Users specify automated and nonautomated resource needs (access controls, regulatory restrictions, interface requirements, and so on) and how they want to have them addressed in the new system.
The IS auditor should verify that requirements are complete, consistent, unambiguous, verifiable, modifiable, testable, and traceable.
IS auditors should determine whether adequate security requirements have been defined for the new system.
Based on the requirements defined in Phase 2, a baseline of specifications for systems is established (application users need not be judiciously involved in this phase). The IS auditor should be allowed to participate in the design phase of the software in an overseeing advisory capacity: Establishing the baseline of specifications is one control used to prevent scope creep during the development phase. Scope creep occurs when new requirements are gradually added and exceed the requirements stated in the baseline of specifications. Although individual new requirements might appear small, they can have a cumulative effect on the cost or time associated with the project (scope creep).
A formal change-control process is established to prevent entry of new requirements into the development process.
Test plans are developed:
Unit (program)
Subsystem (module)
Integration (system)
Interface with other systems
Loading and initializing files
Security, backup, and recovery
The IS auditor is involved to ensure that an adequate system of controls is incorporated into the system specs and test plans, and to ensure that continuous online auditing functions are built into the system.
Programming and testing of the new system occurs in this phase. The tests verify and validate what has been developed. The responsibilities in this phase rest primarily with programmers and systems analysts who are building the system:
Program coding standards are essential to simply and clearly reading and understanding code without requiring review of specifications.
More cohesion (dedication to a single function) and less coupling (interaction with other functions) result in less troubleshooting and software-maintenance effort.
Logic path monitors can provide programmers with clues on logic errors.
Output analyzers validate processing accuracy.
Online programming facilities can increase programming productivity but can also increase risk of inappropriate access. An online programming facility stores the program library on a server, and developers use individual PC workstations to download code to develop, modify, and test.
Test plans identify specific portions of the application that will be tested, as well as the approach to testing:
Start testing with programs or modules, and progress toward testing the entire system.
Testing can be started before the entire system is complete.
Errors in critical modules are found early.
Tests of major functions or processes are conducted early.
Interface errors can be detected sooner.
Testing levels identify the specific level of testing that will occur and are usually based on the size and complexity of the application:
Unit testing
Interface or integration testing
System testing
Recovery testing
Security testing
Stress/volume testing
Performance testing
Final acceptance testing
The organization must include formal project-management techniques in addition to the SDLC to ensure that the software-development project meets the stated objectives. The project-management team should follow the phases of the project life cycle (initiating, planning, executing, controlling, and project closing). The project-management team should develop a detailed plan and report all project activity against the plan, to ensure corrective action where needed. The software-development project should be assigned a project manager who is experienced in the area of software development and has skills associated with managing projects. The successful use of sound project-management techniques will help ensure a successful implementation and minimize the risk of failure because of late delivery, cost overruns, lack of functionality, or poor quality.
During the planning phase, the project-management team should set the time, cost, and scope of the project. In developing the project plan for a software-development project, the project-management team must determine the relative physical size of the software to be developed. In a software project, the planning team and project manager should determine the relative physical size of the software to be developed; this can be used as a guide for allocating and budgeting resources. The project-management team can use function point analysis (FPA) to provide an estimate of the size of an information system based on the number and complexity of a system’s inputs, outputs, and files. Per ISACA, function points (FPs) are computed by first completing a table (see Table 6.2) to determine whether a particular task is simple, average, or complex. There are a total of five FP count values, which include the number of user inputs, user outputs, user inquiries, files, and external interfaces. Table 6.2 shows the formula for computing FP metrics.
After the table entries are completed, the organization can use the count totals and compute them through an algorithm. The results of this computation, which takes into account complexity values specific to the organization, are used to measure cost, schedule, productivity, and quality metrics (that is, productivity = FP/person-month, quality = defects/FP, and cost = $/FP).
When the size of the software-development project is determined, the project team should identify the resources required to complete each task. The project team then should develop a work breakdown structure that identifies specific tasks and the resources assigned those tasks, as well as project milestones and dependencies. The team should create Gantt charts to show timelines, milestones, and dependencies. A Gantt chart is a graphic representation of the timing and duration of the project phases; it typically includes start date, end date, and task duration.
In addition, the project team should perform a program evaluation review technique (PERT).
PERT is the preferred tool for formulating an estimate of development project duration. A PERT chart depicts task, duration, and dependency information. The beginning of each chart starts with the first task, which branches out via a connecting line that contains three estimates:
The first is the most optimistic time for completing the task.
The second is the most likely scenario.
The third is the most pessimistic, or “worst case,” scenario.
The line then is terminated to another node (identifying the start of another task). The chart is completed by connecting predecessor to successor from a network of tasks and connecting lines. The completed chart should depict all tasks coming together at the completion node. All tasks, including milestones, review points, and checkpoints, should be depicted on the chart to correctly estimate the total time associated with the plan. Figure 6.3 depicts an example of a PERT chart.
The calculation of PERT time uses the following formula:
Optimistic + pessimistic + (4 × most likely) / 6
So if you wanted to calculate the PERT time for the project using task 1 terminating at 2, you would apply the formula in this manner:
Task 1 to 2 – (2 + (4 × 5) + 7) / 6 = 4.8
The numbers represent days, so the PERT time for task 1 would be 4.8 days.
After the project teams complete the project plan, they should monitor the progress of the project against the baseline plan using benchmarks, milestones, and deliverables. Any deviations from the plan must be addressed immediately to ensure that the project does not exceed the time and cost estimates.
Many factors can affect the time, cost, and, ultimately, success of a project. And IS auditor can look for some risk indicators in reviewing the project plan and implementation.
The Critical Path Methodology (CPM) is a project-management technique that analyzes successive activities within a project plan to determine the time required to complete a project and which activities are critical in maintaining the schedule. Critical activities have the least amount of flexibility, meaning that the completion time cannot be changed because it would delay the completion of the overall project. The successive critical activities are known as the critical path. The computation of all successive activities along the critical path predicts the required time to complete the project. The critical path of a project can be graphically represented with a diagram showing the relationship of critical activities.
The following are project planning risk indicators:
The organization does not utilize a formal project-management methodology.
The assigned project managers do not have the experience or skill sets to plan and manage software-development projects.
The project has not been reviewed and approved by a steering committee, or does not have senior management support.
Reviews of previous projects indicate that they have not met the business need or that they exceed time and cost estimates.
In conjunction with a formal SDLC and project-management activities, the organization must implement change-management processes, which include change-control procedures both for software-development projects and for the production environment. The change-management process, usually facilitated by the change-control board (CCB), reviews all changes associated with the software-development project. The CCB has the authority to accept, deny, or postpone a requested change. The change-management process ensures that any deviations from the original requirements are approved before being added to the project. This process ensures that all changes meet the needs of the organization and that any additional resources (time, money, or personnel) are balanced against the existing project requirements and approved resources. In addition, the change-management process provides a formal environment for the documentation of changes and the decision process.
A change request can generally be submitted by anyone associated with the development project, including the end users. The submitter of the change request should specify the change as well as the justification for the change. The users of the system should be part of the approval process. User approvals of program changes ensure that changes are correct as specified by the user and that they are authorized.
The presence of a change-management process ensures that subject matter experts (organization managers, IT management, security, and so on) are aware of proposed changes and their impact on current resources and the IT environment.
The organization should implement quality control (QC) procedures to ensure that proper testing is performed through the development life cycle. The QC team is responsible for conducting code reviews and tests to ensure that software is free of defects and meets user expectations. Unit, module, and regression testing ensure that the specific unit or module is complete, performs as expected, and meets requirements. Regression testing should be required for all changes introduced into the system, whether in development or in production. The purpose of regression testing is to ensure that the change introduced does not negatively impact the system as a whole.
After development, testing, and implementation have been completed and the new system is part of the production environment, a formal post-implementation review should be performed. When reviewing an organization’s systems-development process, the auditor should first compare established formal standards to actual observed procedures. The IS auditor should carefully review the functional requirements to ensure that the project objectives and requirements were met. The IS auditor should review the functional requirements and, based on the evidence found, perform other testing to confirm that the necessary controls and functionality are in place. The development of test transactions also can be performed, if necessary. During the review the IS auditor should look at system utilization, end user satisfaction with the system, and error logs to determine whether there are resource or operating problems. A system fails to meet the needs of the business and users most commonly because of inadequate user participation during the systems-requirements definition.
If the organization determined the cost benefit or performed ROI assessments as part of the feasibility study, the IS auditor should ensure that the metrics associated with the production system are being measured, analyzed, and reported. Overall, the post-implementation review should determine whether the development project achieved stated objectives and whether the process of development was performed in an efficient and effective manner. In addition, the post-implementation review should allow the organization to identify areas of improvement through lessons learned. When these improvements are implemented, they improve the overall capability and maturity of the organization’s software-development process, thus maximizing benefits and reducing costs and risks.
The IT organization is responsible for ensuring that the acquired solutions meet the objectives of the organization and reduce risk to the organization through the use of formal acquisition and implementation processes. The IS auditor should assess this process by reviewing control issues regarding acquisition and implementation. A comprehensive acquisition process should include clearly defined procedures for requirements gathering, feasibility studies, proposal creation, and vendor selection. Implementation of the applications should follow a documented system life-cycle development process (SDLC), as well as defined project-management and change-control procedures.
The primary concern for an organization is to ensure that projects are consistently managed through a formal documented process. This process includes the SDLC, project management, change control, and policies/procedures aligned with the strategic plan. The introduction of new software, whether developed in-house or acquired, can affect systems and services if not implemented properly. All software should follow rigorous testing procedures and meet existing standards before being put into production. Before being added to the production environment, the software should undergo vulnerability testing to validate controls and to document and correct any deficiencies. The software should be added to existing procedures (backup, vulnerability testing, and patch/upgrade schedules) to ensure continued confidentiality, integrity, and availability of the software and its associated data. If an IS auditor observes that an IS department fails to use formal documented methodologies, policies, and standards, the auditor should at least document the informal standards and policies, and test for compliance. Furthermore, the IS auditor should recommend to management that formal documented policies be developed and implemented.
The acquisition of software is driven by business requirements. The IT organization should have clearly defined acquisition procedures, including procedures for performing the feasibility study and requirements gathering. In conjunction with managers, the IT organization should analyze the requirements and create the feasibility study. This study helps determine whether the organization will “buy” or “build” the software to meet the needs of the organization. Although software acquisition is not part of the SDLC, it should have a formal documented process. According to ISACA, the project team, technical support staff, and key users should be asked to write a request for proposal (RFP). This RFP should be sent to a variety of vendors, and their responses (proposals) should be reviewed to determine which vendor’s products offer the best solution.
After the vendors respond to the RFP with their proposals, the project team should review them for completeness and determine whether a single vendor can be chosen. If a vendor cannot be chosen based on the proposals, the project team should narrow the list to two or three vendors through a suitable methodology. This methodology should ensure that the same criteria are used in evaluating the vendors. This might require the project team to obtain additional information from the vendors (additional documentation or demonstrations) and to use a scoring methodology to determine which vendor is the best fit for the organization.
After the project team makes the selection, it must negotiate and sign a contract. Per ISACA, the contract should include the following items:
A specific description of deliverables and their costs.
Commitment dates for deliverables.
Commitments for the delivery of documentation, fixes, upgrades, new release notifications, and training.
Allowance for a software escrow agreement, if the deliverables do not include source code. (A clause for requiring source code escrow in an application vendor agreement is important to ensure that the source code remains available even if the application vendor goes out of business.)
Description of the support to be provided during installation.
Provision for a reasonable acceptance testing period, before the commitment to purchase is made.
Allowance for changes to be made by the purchasing company.
A maintenance agreement.
Allowance for copying software for use in business continuity efforts and for test purposes.
Payment schedule linked to actual delivery dates.
As an IS auditor, you should look for evidence of a structured approach to software acquisition. This includes acquisition procedures, defines the process for feasibility studies, and outlines a suitable methodology for the evaluation of proposals. After the software is acquired, there should be a project-management and change-control process to implement the software. All software acquired should be added to existing maintenance contracts, or a new maintenance contract should be acquired. Before being introduced into the production environment, the software should be tested according to written test plans, and responsibilities for the production software should be clearly defined. The software should be secured (physical, logical) and added to the business continuity plan.
As stated in an earlier section, the change-management process ensures that any deviations from the original requirements are approved (or denied) and, if approved, are added to the project. There is one exception to the normal change-management process: the emergency change request used to correct immediate system problems that might affect critical services or processing. The difference between emergency changes and regular change requests is that the emergency change is corrected immediately and the change request (with supporting documentation) is completed after the fact. The organization should use emergency IDs to enable programmers and analysts to log in to the system and correct emergency problems. These IDs should be monitored and logged, and used only in the event of any emergency. The code associated with emergency changes should be stored in an emergency library until the change-control board has reviewed the request. When complete, the code should be moved to the production library. The IS auditor should review emergency change requests as well as the program change log to ensure that only authorized program changes were made.
The IS auditor should review the application system to ensure the control aspects of the system development and implementation process meet the needs of the users as well as management objectives. IS auditors performing this audit should be independent and should not be involved in the system-development process. The IS auditor should assess whether the prescribed project management, SDLC, and change-management processes were followed. The IS auditor should ensure that proper testing procedures were used to validate user requirements and that the system’s internal controls are working as intended.
In addition to ensuring that systems within the application architecture meet the requirements and objectives of the organization, the IS auditor should review them to ensure that controls protect the confidentiality, integrity, and availability of both applications and data. In reviewing the application architecture, the application controls should ensure that information stored, delivered, and processed by applications is complete, accurate, and authorized. In developing control objectives, the IS auditor should keep in mind the following control categories:
Security
Input
Processing
Output
Databases
Backup and recovery
Section AI5 of COBIT (“Install and Accredit Systems”) provides specific activities that an IS auditor should perform to ensure the effectiveness, efficiency, confidentiality, integrity, availability, compliance, and reliability of the IT system. Accreditation is a process by which an organization, through internal or third parties, IT services, or systems ensures adequate security and control exist. The COBIT activities associated with this objective are listed next.
The IS auditor should obtain the following:
Organizational policies and procedures relating to system-development life cycle planning
IT policies and procedures relating to security policies and committees; systems-development life cycle planning; systems-development testing procedures for programs, unit, and system test plans; training of users; migration of systems from test to production; and quality assurance and training
System-development life cycle plan and schedule, and system-development life cycle programming standards, including the following:
Change-request process
Sample system-development effort status reports
Post-implementation reports from earlier developmental efforts
In reviewing this information, the IS auditor should consider whether the following are true:
Policies and procedures relating to the system-development life cycle process exist.
A formal system-development life cycle methodology is in place for system installation and accreditation, including, but not limited to, a phased approach of training, performance sizing, conversion plan, testing of programs, groups of programs (units) and the total system, a parallel or prototype test plan, acceptance testing, security testing and accreditation, operational testing, change controls, and implementation and post-implementation review and modification.
User training as part of each developmental effort is occurring.
Program and system controls are consistent with security standards of the organization and IT policies, procedures, and standards.
Various development, test, and production libraries exist for in-process systems.
Predetermined criteria exist for testing success, failure, and termination of further efforts.
The quality-assurance process includes independent migration of development into production libraries and completeness of required user and operations groups’ acceptance.
Test plans for simulation of volumes, processing intervals, and output availability, installation, and accreditation are part of the process.
A training program associated with a sample of several system-development efforts contains the difference from the prior system, including changes affecting input, entry, processing, scheduling, distribution, interfaces to other systems, and errors and error resolution.
Automated tools optimize systems developed, once in production, and these tools are being used for efficiency opportunities.
Problem resolution is occurring, relating to less than optimal performance.
User involvement and formal approval exist at each phase of the system-development process.
A test plan is in place for programs, units, systems (including parallel or prototype), conversion, and implementation and post-implementation review.
Appropriate consistency is being maintained with security and internal control standards.
Appropriate data-conversion tasks and schedules exist.
Testing occurs independently from development, modification, or maintenance of the system.
Formal user acceptance of system functionality, security, integrity, and remaining risk has been obtained.
Operations manuals include procedures for scheduling, running, restoring/restarting, backing up/backing out, and handling error resolution.
Production libraries are physically and logically separate from development or test libraries.
The organization should ensure quality during the SDLC through testing. Testing should take place at a variety of levels and degrees, but the testing process should follow an accepted methodology. The testing process can be either a bottom-up or a top-down approach. Bottom-up testing starts at the smallest level of the application (modules and components) and works up through the application until full system testing is completed. The advantage of bottom-up testing is that it provides the capability to test single modules or components before the system is complete and allows for the early detection of errors. Top-down testing is usually used in RAD or prototype development and provides the capability to test complete functions within the system. It also allows for the early correction of interface errors. The approach for testing should include the following:
Development of a test plan—. Should include specific information on testing (I/O tests, length of test, expected results).
Testing—. Utilizes personnel and testing software, and then provides testing reports that compare actual results against expected results. Testing results remain part of the system documentation throughout the SDLC.
Defect management—. Defects are logged and corrected. Test plans are revised, if required, and testing continues until the tests produce acceptable results.
In addition to testing, the quality-assurance activities include ensuring that the processes associated with the SDLC meet prescribed standards. These standards can include documentation, coding, and management standards. The IS auditor should ensure that all activities associated with the SDLC meet the quality-assurance standards of the organization.
To ensure that applications function as expected, meet the requirements of the organization, and implement proper controls, they must be tested. These tests ensure that both the modules and the entire system function as designed and that individual modules will not malfunction, negatively affecting the system. Whenever an application is modified, the entire program, including any interface systems with other applications or systems, should be tested to determine the full impact of the change. The following are the general testing levels:
Unit testing—. Used for testing individual modules, and tests the control structure and design of the module. Unit testing pertains to components within a system; system testing pertains to interfaces between application programs.
Interface/integration testing—. Used for testing modules that pass data between them. These test are used to validate the interchange of data and the connection among multiple system components.
System testing—. Used for testing all components of the system, and usually comprised of a series of tests. System testing is typically performed in a nonproduction environment by a test team.
Final acceptance testing—. Used to test two areas of quality assurance. Quality assurance testing (QAT) tests the technical functions of the system, and user acceptance testing (UAT) tests the functional areas of the system. These tests are generally performed independently from one another because they have different objectives.
It is important for an IS auditor to know the testing levels as well as the different types of tests that can be performed. As part of an audit, the auditor should review test results that dictate a specific type of test to be performed, to ensure that controls are operating per the specification. As an example, you might observe that individual modules of a system perform correctly during development testing. You would then inform management of the positive results and recommend further comprehensive integration testing. Testing at different levels and with separate testing elements ensures that the system meets the detailed requirements, does not malfunction, and meets the needs of the users. The following are some of the specific types of tests that can be performed within the testing levels:
Whitebox testing—. Logical paths through the software are tested using test cases that exercise specific sets of conditions and loops. Whitebox testing is used to examine the internal structure of an application module during normal unit testing.
Blackbox testing—. This testing examines an aspect of the system with regard to the internal logical structure of the software. As an example of blackbox testing, the tester might know the inputs and expected outputs, but not system logic that derives the outputs. Whereas a whitebox test is appropriate for application unit testing, blackbox testing is used for dynamically testing software modules.
Regression testing—. A portion of the test scenario is rerun to ensure that changes or corrections have not introduced new errors, that bugs have been fixed, and that the changes do not adversely affect existing system modules. Regression testing should use data from previous tests to obtain accurate conclusions regarding the effects of changes or corrections to a program, and to ensure that those changes and corrections have not introduced new errors.