Footnotes
Preface
2 Fundamentals of Testing
1. The actual behavior is identified while executing the test or during use of the system.
2. The expected behavior is defined in the specifications or requirements.
3. This relates to dynamic testing (see chapter 5). In static testing (see chapter 4), the test object is not executed.
4. It is not possible to prove correct implementation of the requirements. We can only reduce the risk of serious bugs remaining in the program by testing.
5. If a thorough test finds few or no failures, confidence in the product will increase.
6. The cost must include all aspects of a failure, even the possible cost of bad publicity, litigation, etc., and not just the cost of correction, retesting, and distribution.
7. The general V-model will be referred to as the general model to make sure it is not confused with the German V-model, referred to as just V-model.
8. Royce did not call his model a waterfall model. He said in his paper, “Unfortunately, for the process illustrated, the design iterations are never confined to the successive steps.”
9. Another term is test end criteria.
10. In the preceding paragraph, it is unclear what happens at exactly 25.000.
11. ISTQB calls this phase “Evaluation of test exit criteria and Reporting.”
12. See [URL: ACM Ethics] and [URL: IEEE Ethics]. The guidelines listed here are from the ISTQB curriculum.
3 Testing in the Software Life Cycle
1. The V-model is used in many different versions. The names and the number of levels vary in literature and the enterprises using it.
2. To validate: to affirm, to declare as valid, to check if something is valid.
3. To verify: to prove, to inspect.
4. The so-called W-model (see [Spillner 00]) is a more detailed model that explicitly shows this parallelism of development and testing.
5. Actually, there is a defect in this program: Discount calculation for >= 5 is not reachable. The defect is used when explaining the use of white box analysis in chapter 5.
6. Floating point numbers should not be directly compared, as there may be imprecise rounding. As the result for price can be less than 12900.00, the absolute value of the difference of “price” and 12900.00 must be evaluated.
7. Sometimes, two programmers work together, each of them testing the components that their colleague has developed. This is called buddy testing or code swaps.
8. Depending on the compiler, data type errors can be detected during the compiling process.
9. The opportunity to use these types of checks on a component level instead of during a system test is not often exploited. This leads to efficiency problems only becoming visible shortly before the planned release date. Such problems can then only be corrected or attenuated at significant cost.
10. This is a serious flaw because 60 to 80% of the code often is never executed—a perfect hideout for bugs.
11. Special integration strategies can be followed for object-oriented, distributed, and real-time systems (see [Winter 98], [Bashir 99], [Binder 99]).
12. The customer (who has ordered and paid for the system) and the user (who uses the system) can be different groups of people or organizations with their own specific interests and requirements for the system.
13. Sometimes the acceptance test consists of two cycles: the first in the system test environment, the second in the customer’s environment.
14. This verifies that the system complies with the needs of the system administrators.
15. This aspect is not shown in the general V-model. Only more modern life cycle models show iterations explicitly (see [Jacobson 99], [Beck 00], [Beedle 01]).
16. This is regrettably also true for functional requirements. The “of course the system has to do X” implicit requirement is one of the main problems for testing.
17. This is a regression test in a broader sense, where changes also means new functionality (see the glossary].
4 Static Test
1. Some of these rules only apply for some of the review techniques described in [IEEE 1028].
2. See section 6.6.3: Defects of severity class 2 and 3 can be seen as major defects and class 4 and 5 as minor defects.
3. In [ISO 9000] the management review is defined in a more narrow way as “a formal evaluation by top management of the status and adequacy of the quality system in relation to the quality policy and objectives.”
4. Also called “structured walkthrough”
5. According to [IEEE 1028], the participants should receive the documents in advance and should have prepared for the meeting.
6. Often, reviewers are called inspectors. [IEEE 1028] calls the presenting reviewer the reader.
7. In [IEEE 1028], this also applies to inspection.
8. Unified Modeling Language [URL: UML]
9. HyperText Markup Language [URL: HTML]
10. Extensible Markup Language [URL: XML]
11. The original formula is v(G) = e - n + 2p, where p is the number of connected program parts. We use p=1 because there is only one part that is analyzed.
12. This means all complete linearly independent paths.
5 Dynamic Analysis – Test Design Techniques
1. In contrast to stubs, with their rudimental functionality, the →dummy or →mock-up offers additional functionality, often near the final functionality for testing purposes. A mock-up usually has more functionality than a dummy.
2. Black box techniques are also called requirements-based testing techniques
3. White box techniques are sometimes called glass box or open box techniques because it is impossible to see through a white box. However, these terms are not used often.
4. A more correct term would be equivalence classes for invalid values instead of invalid equivalence class because the equivalence class itself is not invalid, only the values of this class, referring to the specified input.
5. If, and which, incorrect values are found by the compiler depends on the chosen programming language and the compiler and runtime system chosen. This may happen when calling the test driver. In our example we assume that the compiler does not recognize incorrect parameter values. Thus, their processing must be checked during dynamic testing.
6. See section 4.2.5 in [Bath 08]. The pairwise combination test method is described in this book in section 5.1.4.
7. It is sometimes useful to combine representatives of invalid equivalence classes to produce additional test cases, thus provoking further failures.
8. Floating-point numbers are part of the equivalence class NaN. See table 5-5 for designing equivalence classes for integer number values.
9. The ideal case is that the identified classes (like equivalence classes in mathematics) are not overlapping (disjoint). This should be strived for, but it’s not guaranteed by the partitioning technique.
10. If the programmer has written if (x ≤ 24999), there will be no semantic difference from if (x < 25000). However, the boundary values determined from analyzing the specification (24999, 25000, and 25001) do not test the implemented statement if (x ≤ 24999) completely. Incorrectly implementing if (x == 24999) would give the same result (true, false, false) for the three values. A code review could in this case find the discrepancy between specification and code.
11. For the test cases, 0.01 was assumed to be precise enough.
12. Remark: A test with 0.00 for the base price is reasonable, but it should be done in system testing because for this input value, calculate price() is not necessarily responsible for processing it.
13. The dependence between the number of extras and extra price (if no extras are given, a price should not be displayed) cannot be checked through equivalence partitioning or boundary value analysis. This requires the use of cause-and-effect analysis (see section 5.1.4 and [Myers 79]).
14. In order to exactly check the rounding precision, values like, for example, 0.005 are needed.
15. And definitely for programming, too.
16. The maximum price for extra items cannot be specified exactly because the dependence between the number of extras and the total price cannot be considered. We used the value 25 × 750 = 18750. An extra price of 0 was not included as a further boundary value because the dependency of the number of extras and the total value of the extras cannot be checked with equivalence class partitioning or boundary value analysis.
17. The precision to be chosen depends on the specified problem.
18. Calling top and pop in the state “empty” have not been specified in the diagram (fig 5-3). This was done on purpose. They will first be taken into account in the extended state transition tree (see figure 5-5).
19. To keep the test effort small, the maximum height of the stack should be not too high because the push function must be called a corresponding number of times to get to the “full” state.
20. The following test cases are simplified to make them easy to understand.
21. The two-staged menu choice is seen here as one action.
22. Note: This is not a complete description of a real automated teller machine but just an example to illustrate the technique.
23. Or even only for every valid equivalence class.
24. ISTQB [URL: ISTQB] is defining a model-based testing add-on to the Foundation Level syllabus.
25. Also called simple condition test or coverage.
26. Also called minimal multicondition test/coverage, modified multiple condition test/coverage, or MC/DC.
27. In the ISTQB Certified Tester syllabus, condition test and multiple condition testing are mentioned only as examples of further structure-based techniques. These two techniques, as well as condition determination testing, are described here anyway because they are effective techniques for testing conditions.
28. Path testing is not mentioned in the ISTQB Certified Tester Foundation Level syllabus. It is described here because it can be seen as a “further step” in statement and branch coverage and because the term is often misunderstood.
29. There exist other techniques not described in this book. This applies especially to integration testing, testing of distributed applications, and testing of real-time and embedded systems. Some of these techniques are part of the Advanced Level Tester certification scheme. More information can be found in [Bath 08].
6 Test Management
1. The term tester is often also used as generic term for all the previously mentioned roles.
2. A detailed description of the listed points in IEEE 829-1998 can be found in Appendix A. The new standard [IEEE 829-2008] shows an outline for a master test plan and a level test plan. IEEE Standard 1012 ([IEEE 1012]) gives another reference structure for a verification and validation plan. This standard can be used for planning the test strategy for more complex projects.
3. A detailed discussion can also be found in [Pol 98] and [Pol 02].
4. There are different methods to assess the maturity of software development processes. More information can be found in the ISTQB Advanced Test Manager syllabus [URL: ISTQB].
5. Medical devices and other safety-critical applications require certification by regulation authorities like, for example, the FDA [URL: FDA]. Such certification follows standards, which require a certain level of test documentation.
6. Such recommendations contain a lot of heuristics and experience-based knowledge.
7. Where defects have been found before, more defects can usually be found! Defects are symptoms of further problems. For defect-prone areas, it is sensible to add extra tests in the following test cycles.
8. An analytical standard method for this is failure mode and effects analysis (FMEA).
9. A spreadsheet-based method for estimating risks or risk classes can be found at [URL: Schaefer].
10. This is the case when the applied test tools automatically provide such data.
11. Incident management is called →defect management in the ISTQB advanced test manager syllabus
12. Creating an incident report may of course also be useful if the tester caused the problem; for example, if the problem calls for further analyses. In this case, the incident will be directed to the tester and not to the developers.
13. This discussion focuses on communication between testers and developers instead of users.
7 Test Tools
1. Commercial tools often support several activities or phases and may be assigned to several tool classes.
2. Many suppliers just refer to a general hotline after purchase.
A Test Plan According to IEEE Standard 829-1998
1. The latest version of the IEEE 829-2008 standard [IEEE 829-2008] differentiates between Master Test Plan and Level Test Plan. An existing test plan created according to IEEE 829-1998 can be converted to conform to an IEEE 829-2008 master test plan—for example, by using an appropriate table of cross-references. However, the 2011 syllabus still cites IEEE 829-1998 as relevant for the Certified Tester – Foundation Level examination.
2. Release is a management decision that the tested product is seen as “ready.”
3. To deliver may also mean to send the test object to the next test level.
4. The standard’s section 2.1. actually contains more details. In order to shorten it and ease the overview, only details for the development process are given here.
5. These eight topics are cited from IEEE 829-2008.
C Exercises
1. These exercises are designed to help you make sure you have understood the terms and processes described in the book and that you can recognize and describe them. Answers to the questions can be found in the relevant chapters and are not listed separately. The questions listed here are not designed to directly prepare you for the ISTQB examination. To prepare for the ISTQB exam, please refer to the mock exam shown at [URL: ISTQB].