4
Test, QA or IV&V Teams
Various terminologies are used to describe the teams in charge of verifying – or validating – the proper functioning of software and systems that make up a system-of-systems.
The term QA (Quality Assurance) describes “a part of quality management, focused on providing evidence that quality requirements will be met”. A QA team will therefore be the people in charge of monitoring manufacturing and manufacturing processes. This term seems incorrect to designate people in charge of software testing because on a project all the participants are actors in the supply of quality deliverables, and there is no notion of assurance in the term quality assurance: no one ensures – in the sense of an insurance company – the quality of the software or component. The activities of the QA teams are mainly the verification of the processes which, if correctly followed, should guarantee the quality, unlike the testing activities which will focus on a functional and/or technical validation of the application. An interesting alternative to “quality assurance” would be “quality assistance” where the members of these teams would assist the production teams in improving the quality of their achievements..
We have the term “receipt” which includes the verification and validation activities of the components before their delivery. According to the organizations, acceptance activities will be called factory acceptance, VAPO (Verification of Aptitude for Proper Operation), VSR (Verification of Service Regularity), FAI (First Article Inspection), etc. The term “receipt” does not have an adequate English translation and does not cover a defined perimeter accepted by all. The origin of the term “recipe” could come from the English “receipt” in the sense of receiving the application.
QC (Quality Control) focuses on verifying compliance with processes and the achievement of quality requirements. We are here in a subset of quality assurance – which deals with product quality – to focus on inspecting the processes that will ensure quality.
The term IV&V (Independent Verification and Validation) represents – for example, in aeronautics or space – the teams in charge of verifying that the processes have been followed correctly, the requirements developed and the components designed and installed according to the rules of the art. The “I” in IV&V indicates that the teams must be independent of the production teams. This IV&V approach can be more general than simple testing and include inspection and analysis activities (see also IADT) throughout the product design.
Other terms such as “qualification” and “certification” are also used in the field of systems-of-systems. The field of qualification applies rather to the verification of people’s skills, while certification applies to the supply – by a competent authority – of a document (the certificate) allowing the use of a product (e.g. certificate of airworthiness allowing an aircraft to fly).
In the context of this work, we consider “test teams” as all the people in charge of the verification and validation of products – mainly related to software – used in systems-of-systems and in systems with leading software. This may include people in charge of document reviews (reviewers, experts, authors, etc.), code-related deliverables (general specifications, detailed specifications, architecture diagrams, flowcharts, installation or usage procedures, test reports, etc.), without this list being exhaustive. The common objective of the members of the “test teams” is to find defects as early as possible and to provide information on the level of quality achieved.
4.1. Need for a test team
The need for a test team varies depending on the chosen development cycle and the level of testing or time in the development cycle. At the level of systems-of-systems projects, the cycles are mainly sequential and the need for multiple test teams, at least one per level, is real. The main concern with this solution is that defects may not be detected until late, which will increase costs and delays.
For systems developed using Agile or DevOps methods, the principle of detection as early as possible involves putting the testers as close as possible to the developers. This does not prevent adding teams of testers dedicated to the execution of tests specific to systems-of-systems (e.g. system integration tests, operational tests, etc.).
As mentioned earlier, the skills needed for testers and the organization of test teams will vary across development cycles. In a sequential cycle, it is possible to have task specialization and task execution in sequence, as proposed by Taylorism. In an iterative and incremental design, and even more in a DevOps-type development mode, it is necessary that the testers’ activities cover both static verification activities (static reviews and analyses) and dynamic test activities (tests white box and maintainability during unit testing, functional testing and technical testing – performance, security, etc. – at other test levels) and can act almost simultaneously on multiple environments (integration of components, integration of systems, systems and acceptance, etc.).
In each of the two cases, the skills and knowledge required will be necessary within the project, in order to detect the defects and to validate the operation of the system-of-systems.
4.2. Characteristics of a good test team
Regardless of the term used to refer to verification and validation teams, certain elements are necessary for this team to carry out its work successfully:
– Sufficient size: having a test strategy is necessary, but if our test teams do not contain enough people to perform all the defined tasks, the strategy will never be able to achieve its objectives on time. Undersized teams often move from project to project, crisis to crisis, sometimes with enough time to test and add value, but such a team cannot be seen as an important part of the organization’s strategy as part of the quality.
– Adequate resources: a team without resources can – at best – only do manual activities. The complexity of the applications increases the combinatorial explosion of the tests to be considered. The growing number of deliveries over a limited period requires more frequent retesting or non-regression testing over a wider functional coverage. It will therefore be necessary to have sufficient resources, whether we are talking about personnel, automatic execution tools or execution time. The absence of one or more of these resources in sufficient number will mean that the tests will not be able to cover all the combinations and that defects will remain in the systems composing the system-of-systems.
– Well trained: the number of people is not enough; individuals must have adequate training for the project. An expert team brings added value 10 times greater than a weak team. A strong team brings about two and a half times more value than a standard team. Therefore, by training our testers in a targeted way, we can more than double their contribution. It is therefore important to have a training budget and to ensure that the training is well focused.
– Well equipped: a team with the right knowledge and skills will never be able to achieve the defined objectives if it does not have the adequate resources (e.g. test environments, test tools, test data, tools defect tracking or test management or requirements management, powerful workstations, high-speed access, etc.). Certainly, testing resources are expensive, but those costs can be pegged to all the projects that the test team handles, as well as the entire duration of the projects. These costs must also be related to the costs of failures that are avoided.
– Knowledgeable: testing provides information, but test teams also need information. To produce useful information in a timely manner, it is necessary for test teams to obtain information early: testers must understand the objectives, obtain test environments and test data, adapt activities to changes requirements or priorities, etc. Any delay in providing information to the test team will cause delays in providing information out of the test processes, and test processes are often on the critical path of the project.
– Independent: the information provided by the test teams may not correspond to the expectations of the development teams or to the wishes of the stakeholders. The number of defects to be corrected may prove to be too high compared to the current capacity of the team, the planned delivery dates may prove to be too ambitious for the level of quality desired by customers, etc. If the testers are members of the development team, they might have their recommendations ignored so as not to penalize the development team. It is therefore recommended, depending on the test levels, to have different levels of independence: more independent closer to delivery.
– Respected: this last element is strongly linked to the previous four elements. When a test team is respected, the above items will be easy for the test manager to obtain. If, on the other hand, the above elements are not present, the test team will not be able to provide relevant information in a timely manner and therefore it will not be respected. Respect is not something given, but rather something earned. One way to earn respect is to interact with other stakeholders with respect and professionalism. By focusing on ways the test team can add value to the organization, the test manager can increase respect for the test team.
4.3. Ideal test team profile
There is no ideal profile for a test team or for a tester. That said, since the goal of a test team is to be effective and efficient in performing their tasks, the sum of the profiles should cover all activities and areas in which the team is called upon to work. This includes:
– Technical computer skills, to be able to manage test environments, the addition of new equipment or tools or contributors, to independently manage backups and restores, manage queries and databases, etc. These technical skills allow the testing team to get by with minimal impact on the organization’s support teams.
– Methodological skills in testing, to know which test methods to use according to the use cases, to define the right test data to use (valid or invalid data), to be able to explain the interest of each test technique and to measure the percentage of progress or completeness.
– Professional and business skills, to understand the business documents provided, identify nominal use cases and exceptional cases; these business skills will be very useful in determining the significance of the impacts of each anomaly.
– Skills in testing tools if such tools are used. This may involve development experience and skill if TDD tools or capture/replay tools are used.
– Management skills, which will be necessary for the test manager (or the person who will have the role of test manager). These skills include communication and activity monitoring skills, as well as synthesis and prioritization to provide relevant reporting information.
– Software development skills, to know when to interact on which deliverables, without acting at cross purposes with respect to the development activities – hardware or software – on the project.
– Etc.
The list of skills will vary depending on the levels, the components or software to be tested, as well as the test objectives that the team must address. A team in charge of unit testing a management software within a banking system-of-systems will need different skills from a similar test team for a system-of-systems in the aeronautical field.
4.4. Team evaluation
It is very rare that a single person brings together all the necessary skills within a test team. Generally, the necessary skills are distributed among the people who make up the team.
One thing to keep in mind is that a team suitable for one project may not be suitable for another project. The evaluation of the skills of the team can therefore never be done if we do not have – beforehand – an assessment of the needs for the project and probably for the possible evolutions of the project.
The test manager will assess the skills of each member of their team to have a complete view of the strengths and weaknesses of the test team. Depending on the results of the assessment, they will propose skills improvement actions or select testers with specific skills, business or technical.
It should be noted that training and skills improvement have two complementary objectives: acquiring a qualification to perform a job and raising the skills of test engineers so that they feel that the company is investing in them.
4.4.1. Skills assessment table
A solution to assess the adequacy of the test teams with the needs of the project is to create a table listing the needs of the project with the level of knowledge required and to put it in front of the skills – and the level of knowledge – of the team members.
We generally consider the following sections:
– formal education, certifications and continuing education;
– experience in the position or function, functional experience, IT;
– professional and relational skills;
– general testing skills, knowledge of standards and methods;
– skills in test design and test techniques;
– skills in static testing, reviews and static analysis;
– skills and experience in the design and execution of automated tests;
– knowledge of configuration, version and anomaly management;
– experience in manual execution of tests, and exploratory tests;
– experience in KPI measurement and test reporting.
For each section, we define the level of knowledge (none, beginner, intermediate, expert) as well as the need (desired or required). This gives us a table like the one in Figure 4.1.
We then evaluate the members of the test team – or the people approached – based on the same criteria.
Figure 4.1 Example of skills assessment table © RBCS
From this, we can compare the results and identify:
– areas where a need is identified, and no resource has the required level of knowledge;
– the maximum and average knowledge in the team in order to identify whether the information resides with one person (risk in the event of the unavailability of this person) or is spread over several members of the team.
In the event that needs are identified, it is possible to use the table to discuss with human resources and purchasing to determine whether it is:
– possible to train people on the test team in the areas identified; this solution is the most profitable if the project is planned to last;
– preferable to use subcontracting (independent or CDD), interesting if the need is of limited duration;
– more interesting to use a subcontracting company (ESN) to carry out the tasks requiring these skills.
Whichever solution is chosen from the last two, it is necessary to include aspects of skills transfer to other team members in the contracts to ensure that information or knowledge is not lost in the end of contract.
If a recruitment is planned, the skills table makes it possible to better define the profiles sought and to compare the summaries received with the needs identified.
4.4.2. Composition
In general, the test team will be composed of, for a given test level:
– a test manager or a person in charge of planning, coordinating level testing tasks, monitoring and reporting level testing activities; in general, a test manager also has an activity of supervising the testers and test analysts of the team, as well as training and coaching responsibilities;
– one or more test analysts, in charge of the design of the tests and optionally their execution;
– one or more technical test analysts;
– one or more test automation engineers, in charge of automating level tests, their execution and the maintenance of automated tests;
– one or more test technicians or functional testers.
Test analyst and technical test analyst activities may include supervision of technicians or automation engineers.
4.4.3. Select, hire and retain
Selecting the right profiles for the test team is not easy because we must select both the best profiles – combining skills, experience and know-how – but obtaining these profiles for an amount that the company can afford.
Once the profile is defined – based on the skills tables – and the advertisements are published with the help of the human resources services, the selection can begin. Depending on the state of the market – favoring the employer or favoring the employee – the number of responses can be high or relatively limited. The test manager will therefore be required to evaluate the proposed summaries, to see their adequacy with the proposed position (identify gaps and overlapping points), carry out interviews and check references. When the profile is selected and the tester accepts the proposal made, it will then be necessary to proceed during the first weeks with “onboarding”, which aims to familiarize the newcomer with the business aspects, the systems and their interrelationships, the organization, the repositories, the ways of doing things for the test team and to ensure that the recruit fits well into the current team.
It is obvious that the process from the initial decision to bring in an additional resource until the new hire is effective is time consuming and expensive. It is therefore crucial that the arrival of the recruit does not generate tensions only within the team. The impact of the departure of an existing member of the team would be costly for the company and harmful for the team (loss of knowledge and cohesion). Maintaining good relationships within the team and between the test team and other teams (developers, business, etc.) is something the test manager should monitor and strive to maintain. If an individual proves harmful to the good of the team – for example, by questioning the qualities of other team members – it will be necessary to take prompt action to avoid a loss of motivation of the other members of the team.
4.5. Test manager
A test manager, like any manager, knows that their objectives will be achieved only by and thanks to their team. Managerial, organizational and human capacities are very important for the test manager to be effective. Among other actions, the test manager must ensure that their test teams are:
– of sufficient size;
– with adequate resources;
– well trained and well equipped;
– well informed;
– respected.
This involves informational roles, both towards the test team – or even the test teams – and towards management and project management. In this second role, the test manager must be the spokesperson and advocate for the test teams. The role of information provider is important throughout the implementation activities – from the design of the tests to their execution – but is essential in the conclusion phases of the tests, when it is necessary to decide whether the product is sufficient quality to proceed to a next step or if it needs to undergo further testing. This decision must also involve the players in the business and the test manager can explain the impact of this or that anomaly.
It also involves a decision-making role such as assigning a tester to a task, selecting a method or tool, having overtime work done, etc. This aspect of decisions can be applied in the form of advice during configuration management activities (e.g. during CCBs), decisions on organization or management, for example related to the quality of the software, its ability to move to the next phase, the probability of occurrence of failures and the possible impacts.
4.5.1. Lead or direct?
There are various ways to manage teams and people: some will tend to say “forward” while others will say “follow me”. Some managers will tend to provide goals and allow their teams to manage to achieve them, while others will tend to micro-manage their teams. Agile methodologies offer solutions that can be difficult to implement in environments where the weight of the hierarchy is important, or when the people making up the team are more junior (or less autonomous).
It is up to us to implement the management model that works for us and our team and our organization. In the framework of systems-of-systems, as the aspects of collaboration between organizations are very important, the qualities of diplomacy, synthesis and persuasion are important.
4.5.2. Evaluate and measure
Throughout a project, the test manager will be required to measure the progress of the tests, to compare it with the planning and to evaluate the level of quality of this progress. These aspects are evaluated in more detail in the following chapters.
However, the test manager will also have to assess:
– the adequacy of the proposed test strategy in relation to the constraints of the project and to propose improvements if necessary;
– the level of quality of the elements at the start of the test phase, whether it concerns the description of the requirements or the user stories, the test data needs, the test cases and the coverage achieved of the techniques of test or risks, depending on the test methodologies chosen;
– the number and criticality of anomalies identified by test level, to assess the need for improvements in the upstream phases of the tests as well as in the current phases of the tests;
– the needs for continuous improvements, either to improve the current coverage of requirements or to improve the test cases according to the anomalies identified later (e.g. in production) and which should have been found in this test phase;
– the needs for reporting, training, coaching etc.
For this, the test manager will need metrics and other KPIs. It will be necessary to pay attention to the fact that when we measure something, it has an impact on these things. We can take multiple examples here where the object of the measurement is in fact diverted and ultimately serves no purpose.
In a forklift factory in a socialist country, productivity was measured by the number of trucks produced per month. This productivity had increased significantly but without impact on sales: these carts were piling up in the yard as they were missing tires. Since the tires were purchased externally, the factory’s productivity managers considered their job done.
Examples also exist in countries where the economy is not controlled: a co-contractor being paid for the supply of cabin sections, they achieved their objectives but by supplying non-cabled sections, without the wiring harnesses required. This implied an increase in activities in the later phases, additional costs and delays.
4.5.3. Recurring questions for test managers
Test managers must constantly ask themselves many questions and find answers to them. Among these questions, we have the following:
– What is the exact level of functional coverage achieved by the test campaign? In the event of changes to the scope (e.g. addition of components), the tests carried out beforehand lose their value because there may be regressions.
– What is the level of completeness achieved in the implementation of the test techniques? Are the techniques correctly and exhaustively implemented?
– For each defect identified, were we able to determine the age of the defect (is it a defect highlighted in old code or a defect in newly written code?), what is the technique that is able to identify it (to compare the effectiveness of the techniques), what is the main root cause (to prevent it from recurring), how could we have identified this defect earlier, how could we have avoided its introduction, etc., to improve testing and development processes?
– For all automated tests that work without identifying any faults, how can we be sure that there are no false negatives?
– How can we aggregate test cases and test data so that tests can be run automatically, without human intervention, with every build or code change?
– Are the test reports clear and understandable?
– Are the risks correctly identified and mitigated (risks related to components or features or functionalities not covered)?
– Are there trends to identify in the number of defects in the backlog?
– What is the average defect correction time? Is it reasonable?
– Are the defects concentrated – grouped – by functionality or by component or are they evenly distributed?
– Is it possible to reduce the duration of the test campaign without reducing its coverage?
– How can we design the test cases at the same time as the PBIs are developed (hint: it is possible)?
– How can we run end-to-end tests across multiple environments, multiple applications and multiple systems?
– How can we manage SSO (Single Sign On) and tests across multiple platforms simultaneously?
– How can we measure the functional coverage rate for each application? Should we limit ourselves to passing cases or should we also deal with non-passing cases?
– How can we measure the technical coverage achieved by the tests for each application?
– How can we measure test method coverage for each application? What are the most effective test methods?
– How can we determine the cost of correcting defects found early and those discovered late? Are the data reliable and reproducible?
– Are the deliverables provided by the teams clear, unambiguous, understandable? Among the deliverables: anomaly reports, test campaign execution reports, test plans or approach, etc.
– Do the test engineers have the technical, methodological and business knowledge necessary to carry out their activities? What training should be anticipated?
– What techniques or methods should be considered for testing in the future (improvements to be considered for the short, medium and long terms)?
– Will the environments planned allow tests to be executed in a way that will meet the needs of users and the business (in addition to the implementation teams)? Will we be able to properly do end-to-end testing?
This list, of course, is not exhaustive.
4.6. Test analyst
The test analyst is an experienced tester focusing on functional aspects and mastering dynamic and static testing techniques in addition to the functional domain. Meticulous and pragmatic, a test analyst must be able to supervise and lead teams of more junior functional testers.
The knowledge of a test analyst should cover:
– ability to analyze, understand and synthesize in order to properly interface with business experts (e.g. product owners) and technical experts (developers);
– understanding and mastery of test techniques, their advantages and disadvantages, the types of anomalies they can identify;
– ability to combine different test techniques to cover functional needs.
Questions a test analyst should ask include (non-exhaustive list):
– Have the risks identified in this component, this functionality, this system been correctly covered?
– Have the test techniques used been correctly implemented?
– Is the combination correctly covered, according to the risks defined?
– Is the level of coverage of business rules consistent, validated and verified?
– If using decision tables, have the columns been optimized?
– In the case of E2E tests between applications, are the transitions from one application to the next consistent and represented by specific test cases in each application (so that they can be reused by simply changing the data)?
– Do the test cases include verification/validation of the results obtained?
– Do the test cases have setup and teardown phases?
– Can the automated test cases on my workstation be run from another workstation without maintenance?
– Can the test cases be executed without human intervention?
– Are the results of the automated tests reliable?
4.7. Technical test analyst
Technical equivalent of a test analyst, the technical tester is focused on the technical aspects of the test, such as performance and security. Like the test analyst, a technical test analyst must be able to mentor more junior testers to achieve defined technical test objectives.
The range of areas to be covered for a technical test analyst is extremely wide and includes:
– performance tests with the design of scalability profiles, analysis of the results obtained;
– tests on radio networks (including Bluetooth, GSM, 4G and 5G) as well as the impacts in terms of communications between clients (tablets, mobiles, etc.) and servers;
– information access security aspects (e.g. pentest), non-repudiation of transactions;
– aspects related to the adaptation of tests according to the architecture of the components that make up the system-of-systems (see Ford et al. 2021);
– aspects related to accessibility, specifically accessibility for people with disabilities;
– the modularity and interfaces that will need to be tested between each of the elements (modules) whether APIs (for software components) or other means of communication for interactions between subsystems.
We note that it is unlikely to have all this knowledge grouped within a single individual, so the test manager will have to consider calling on several people.
4.8. Test automator
A test automator is a tester who masters automation tools and is able to design test frameworks and integrate several test tools together. The test automator is – for software testing – the equivalent of an experienced analyst-programmer. The test automation engineer must know scripting languages and master operating systems as well as test environments.
The test automation engineers are the counterpart of the developers for the tests. Their objective is, through test automation, to facilitate the execution of test campaigns, to reduce their duration and to guarantee their quality. In fact, they must both have an intimate knowledge of systems and development, as well as be able to translate test cases – functional or not – defined by test analysts into automated test scripts. These scripts should then be able to be played independently (e.g. at night without human intervention) and provide results that are reliable.
Test automation engineers should understand the architecture of the tests envisaged at each test level and how to ensure the traceability of the tests to the components tested (classes, methods, function programs, etc.), to the requirements (or user stories) as well as to functional or technical areas important to the organization.
In the same way, test automation engineers must be able to propose test frameworks that can implement techniques such as keyword-driven testing (KDT), or recursive tests that can be called from a system to another. The objective here is to be able to design a test environment that can integrate several test automation tools and merge the results in a coherent way for test managers and other stakeholders. Of course, the test automation engineers will have to design scripts that are modular, maintainable and understandable because these automated scripts will have to be able to have a lifespan at least equal to the components they test.
4.9. Test technician
A tester at the beginning of their career, a test technician is the equivalent – for software testing – of a programmer. The activity is mainly manual execution of tests designed by others, but its evolution will lead it to focus on its areas of predilection such as functional, technical tests or test automation.
4.10. Choose our testers
The goal for a test team is for the whole to be more effective than the sum of everyone that composes it. This is difficult to achieve if we do not consider the human and relational aspects, in addition to the aspects of technical and/or professional knowledge of the testers. The test manager has a very important role in setting up and leading an effective test team: they must be both manager of the test team, interlocutor of the business and development teams, adept at communication with management, mastering the various development – and test – methodologies and knowing about test tools, test data. It is obvious that a mastery of the upstream and downstream aspects of testing activities, including the various levels of testing, will be welcome.
In addition to technical and business skills, testers must also be comfortable in environments where very little is stable and fixed: applications evolve, deadlines are often very short, pressures are high and activities are repetitive. Pressures come from all sides, from development, customers, hierarchy, etc.
If the test manager insists that overtime be worked without a valid reason and without compensation, the team’s testers will quickly become tired of this lack of recognition.
It is the responsibility of the test manager to protect their testers while giving them the opportunity to express themselves, to coach them without limiting their initiatives, to push them to improve continuously without demotivating them with excessive demands.
HR teams and psychologists can use techniques and/or personality questionnaires (e.g. Belbin) to help organize our teams. The selection of a tester is a very important activity for the test manager: if a person is selected and does not fit the bill, it will negatively influence the team and delay the success of the project.
4.11. Training, certification or experience?
Is it better to have a trained person or a certified person? An experienced tester or someone with a good initial training and a willingness to learn? There is no simple answer: we need a mixture of all of these.
Some training methods, such as those focused on tools, are often limited to how to implement the tool on a fairly limited demonstration project. The complexity of the project, the use of multiple tools and the reporting needs are often overlooked. Other training, such as that leading to certifications, is often simply cramming and of very limited value.
A certification shows that the individual – or their hierarchy – wished to show a level of professionalism in the testing professions. Since test certification programs are funded in whole or in part by the number of certifications sold, it is normal that the level of exams is not too high. A certification alone, especially if it is a basic certification (e.g. foundation level) has no value if it is not based on prior experience. For advanced levels, it is also preferable to pass the certification after having had a few years of experience in the position.
Diversified experience, in various fields, is very important for a tester. Diversification should include both skills in automated testing, on various technical environments (mainframes, networks, minis and micros), skills in test methodologies and even skills in software development. Experience in various fields (e.g. banking, critical security systems, network, retail, etc.) demonstrates the individual’s ability to learn and adapt.
4.12. Hire or subcontract?
The choice to hire or outsource must be seen in the short, medium and long terms, and take into account both a financial view and a more technical view as well as longer-term needs or potential.
The hiring aspect must be seen in both short-term and long-term dimensions. If the objective is to have a team that is quickly operational, the hiring of qualified personnel is necessary. If the objective is to create a multidisciplinary team in charge of quality and testing, it may be necessary to consider hiring more junior people – less competent and cheaper – who have good skills and can be trained. The choice will have an impact on the ability to achieve in the short term and on the training efforts to be implemented in the longer term.
If the assignments or additional activities are one-off and the duration of familiarization – functional or technical – is short, subcontracting may be justified.
If the load forecast shows a constant load over time or if the time for skill development is significant, it may be worth hiring. This will ensure a long-term return on investment and avoid a loss of business knowledge.
Subcontracting must be considered to compensate for a temporary increase in load. On the other hand, it is necessary to make sure to avoid the loss of functional or technical knowledge that can occur when the subcontractors leave. One way to guard against this loss is to define, in the subcontracting contract, a documentation or knowledge transfer phase.
Outsourcing for simple financial needs can be justified at company level, but involves significant risks if applied to positions requiring particular skills (e.g. knowledge of particular or old languages or tools).
4.12.1. Effective subcontracting
Having an effective subcontracting of certain test activities requires that the contracts binding the principal and the subcontractor are defined and that certain principles are implemented, as mentioned by Watts S. Humphrey (1998). The key criteria are:
– technical competence and reliability of the subcontractor;
– ability on the part of the buyer to verify the competence of the subcontractor;
– the contract implies mutual trust;
– quality assurance and audits ensure honest and reliable performance;
– initially the highest priority task is to reach agreement on what is to be produced, how to produce it and the acceptance criteria;
– it is recognized that the operational concept and product requirements will evolve throughout the contract and forecasts will be implemented to manage the vision and load plan.
4.13. Organization of multi-level test teams
In a system-of-systems, where several test teams work in parallel, it is important to avoid performing duplicate work and also to avoid leaving uncovered elements (holes in the racket). This implies a shared global view and good communication between the different teams. The use of common tools can facilitate this communication but must be balanced against the security and confidentiality aspects related to the sharing of the same tool between various co-contractors.
The organization of these tests can be specified in the test strategy of the system-of-systems, or defined from the test activities of each of the systems. The most effective solution requires shared trust between the co-contractors, as well as the provision of test evidence related to each of the requirements for each of the systems and subsystems. This solution makes it possible to limit the tests to be rerun. However, such a level of confidence is not always achievable and some co-contractors consider the components they supply as COTS components that do not require additional testing.
4.13.1. Compliance, strategy and organization
The test strategy of a system-of-systems can be seen as the means of defining the proofs of conformity and of building the conformity of the system-of-system through the conformity of the systems composing it. The organization of the test teams will therefore depend on this test strategy.
The preparation phase of the construction of compliance within the framework of a system-of-systems aims to define, for co-contractors and sub-contractors, what evidence and means of proof are necessary to ensure compliance of components with respect to the requirements that define them. This document, like the test strategy, should be designed and written by the organization in charge of the project management, in order to guarantee the most generic vision possible of the functional needs of the system-of-systems.
Throughout design and construction, the teams in charge of quality and testing will have to provide evidence that compliance with requirements has been achieved. The type of evidence and the level of detail of this will vary depending on the level of testing.
The application of this conformity construction document – of this test strategy – will impact the actions to be carried out at each test level and for each of the components for which compliance must be demonstrated with respect to the requirements. Compliance can be achieved in different ways, through the delivery of documents, proof of reviews, test results, etc. Each of these deliverables will impact one or more levels and one or more teams (even co-contractors).
4.13.2. Unit test teams (UT/CT)
In general, unit tests are performed by developers in the development environment. In the case of TDD (Test-Driven Development), unit tests are designed before the code, by the developers. Testers can participate or advise developers in the design of unit tests.
Depending on the size of the components or unitary elements within a system-of-systems, the complexity of the tests to be implemented, their number and the effort required to execute them will vary greatly. In general, the teams in charge of unit tests are close to the production teams, or even incorporated into these teams. In the case of agile developments within Scrum teams, the “testers” are “developers” like the others.
4.13.3. Integration testing team (IT)
Integration tests can be carried out at several levels within the framework of systems-of-systems, depending on the breakdown of this system-of-systems and its components. If we use an ERP or a CRM (e.g. SAP, Salesforce, etc.), there will be on the one hand the integration of the elements developed with this ERP or this CRM (e.g. new screens, specific settings, etc.) and then the integration of this ERP (or CRM) with other applications – for example, Legacy – via EAI or ETL.
The handover between each team will be defined and will identify precisely what is included in each of the components delivered and integrated: anomalies corrected, new functionalities introduced or modified, etc. It will therefore be necessary to synchronize the configuration management procedures to ensure the completeness of the integration deliveries.
Integration test teams should include both representatives of the development team, technicians – if integrating hardware components – and testers with technical skills in addition to functional skills applicable to that level of integration.
Depending on the level of integration testing – software only or systems or equipment with predominant software – the presence of technicians in charge of maintenance and logistics support teams can be useful.
4.13.4. System test team (SYST)
System testing can be performed at several levels depending on what is included in the “system”. If we are talking about systems or equipment with predominant software, we will be focused on the black-box operation of the equipment and we will need testers who can master the technical, mechatronic and electronic aspects. On the other hand, if the system is only software, these needs related to the hardware aspect will be useless.
System test teams should have knowledge of the functional aspects of the system under test, as well as of the functional interactions of the system with its environment and other systems in the system-of-systems. This allows them to more correctly assess the functional impact of an anomaly. Similarly, knowledge of the means of exchange between systems and the types of data exchanged will also be necessary.
4.13.5. Acceptance testing team (UAT)
Acceptance testing teams are the responsibility of the customer. Generally, their composition is determined contractually.
4.13.6. Technical test teams (TT)
As technical tests can happen at all levels, technical test teams will have to include – in addition to the types of personnel relevant to the level – specialists depending on the type of test – and the quality characteristic – envisaged:
– performance (load increase, load peaks, etc.);
– security (pentest, access authorization, etc.);
– maintainability (readability, comments, etc.);
– portability (modularity, limitations, etc.);
– etc.
The execution of technical tests is often postponed after component tests, integration tests and system tests. The argument put forward is the need to ensure that the functional aspects are correctly mastered before focusing on the technical aspects. This results in:
– a late discovery of technical problems which often leads to a more or less major overhaul of the system, with significant impacts in terms of costs and deadlines;
– problems about delays and bottlenecks in case the planning of activities is not fully mastered.
We recommend involving the technical test teams from the design stage, in order to obtain their recommendations in terms of system architecture and organization, and at all levels of testing in order to verify whether compliance with the technical recommendations, requirements safety or other devices are correctly installed.
4.14. Insourcing and outsourcing challenges
The use of subcontracting is frequent to parallelize activities or reduce costs by using cheaper labor. This cost reduction comes from economies of scale in the case of specialized subcontracting, or the use of resources in countries where labor is cheaper. In any case, the use of outsourcing to execute test activities generates generic challenges, as well as challenges specific to the use of co-located, nearby outsourced or geographically distributed external teams.
Generic outsourcing challenges include:
– loss of skills and historical information transmitted orally between individuals;
– economic dependence on another organization and on the level of quality that this other organization implements;
– creation of potential competitors;
– difficulty of reinternalizing previously outsourced skills.
Specific challenges include:
– the increase in risks related to the creation of foreign competition trained in outsourced applications (i.e. our intellectual capital);
– the difficulty of treating external and internal staff in the same way;
– the objectives of the various teams which may not correspond or even be in conflict;
– information and process security management, especially given the need to exchange confidential data between potentially competing entities.
4.14.1. Internalization and collocation
To avoid the risks associated with geographical spread, it is possible to put all the players – internal and external – within the same premises. In the case of internalization and co-location, it is possible to manage the teams as if they were part of the organization. This involves various aspects:
– Should internal and non-organizational staff be treated the same way?
– Aren’t there potential risks of conflicts of interest between external staff – who report to another organization – and internal staff?
– Are the premises suitable for the number of participants and what will be the relocation costs to be considered?
– Is the security of data and processes properly considered for each type of personnel (internal or external)?
More and more organizations are using service companies (in France currently called ESN for Entreprise de Service Numérique) to handle their needs. If such solutions are interesting at the economic level (we can reduce the costs when there are fewer needs) and at the technical level (we can call on specialized ESNs with the right technical level), the excessive call on a outsourcing seems to be a significant risk of loss of skills and dependencies on people outside the organization.
4.14.2. Near outsourcing
With the reduction of communication costs and the internationalization of the Internet, the use of outsourced subcontracting in the same country or in close countries is becoming more and more frequent. The skill level of outsourced stakeholders is not always easy to assess. Some will consider that the international deployment of a certification scheme should make it possible to reach a sufficient level. This vision of things is true if this certification really makes it possible to ensure that the certified people have understood and assimilated the knowledge and are able to reproduce it. However, if the acquisition of these certifications is seen only as an entry ticket or a badge, without representing a real mastery of complex skills, it is permissible to doubt the effectiveness of certified people.
Close outsourcing, in the same country or in a nearby country or in a close time zone, can be a solution to overcome the lack of manpower or to reduce labor costs. On the other hand, it is necessary to understand that this will increase the risks and the delays: in addition to the risks linked to safety, it is also necessary to take into account the differences in cultures, in the way of working, in the timetables, in public holidays, in to do, etc. It must also be understood that outsourcing actions – near or far – are based on contracts, and not on the objective of success of the system-of-system as a whole. Moreover, whether the test activities are carried out in the organization or elsewhere, the duration and the load of these activities has no reason to decrease – on the contrary – and the only gains would be the possibility of working in parallel or reducing the costs. Of course, the risks related to coordination concerns between co-contractors and the transfer of information between them will increase.
4.14.3. Geographically distant outsourcing
The globalization of world markets, the desire to reduce payroll costs or the distribution of work between co-contractors means that more and more geographical distribution between several places, countries and time zones is becoming usual. A few elements make these implementations more complex and increase the risks:
– geographical distribution over several locations in the same country and in the same time zone makes exchanges between teams more problematic, even if teleconferencing techniques are developing;
– geographical distribution between countries where the vehicular language is different adds the risk of misinterpretation due to translation concerns; in addition, certain local habits (e.g. the fact of not saying “no” to one’s interlocutor) make exchanges more complex;
– a geographical distribution between different time zones forces some participants to get up very early or go to bed very late to participate in joint meetings: what would be the right time for a meeting including participants in San Francisco, Paris and Hong Kong or Sydney?
Of course, these potential concerns add to each other and add to the complexity of the system-of-systems project.