SPECIFICATION-BASED (BLACK-BOX) TECHNIQUES
The main thing about specification-based techniques is that they derive test cases directly from the specification or from some other kind of model of what the system should do. The source of information on which to base testing is known as the ‘test basis‘. If a test basis is well defined and adequately structured we can easily identify test conditions from which test cases can be derived.
The most important point about specification-based techniques is that specifications or models do not (and should not) define how a system should achieve the specified behaviour when it is built; it is a specification of the required (or at least desired) behaviour. One of the hard lessons that software engineers have learned from experience is that it is important to separate the definition of what a system should do (a specification) from the definition of how it should work (a design). This separation allows the two specialist groups (testers for specifications and designers for design) to work independently so that we can later check that they have arrived at the same place, i.e. they have together built a system and tested that it works according to its specification.
If we set up test cases so that we check that desired behaviour actually occurs then we are acting independently of the developers. If they have misunderstood the specification or chosen to change it in some way without telling anyone then their implementation will deliver behaviour that is different from what the model or specification said the system behaviour should be. Our test, based solely on the specification, will therefore fail and we will have uncovered a problem.
Bear in mind that not all systems are defined by a detailed formal specification. In some cases the model we use may be quite informal. If there is no specification at all, the tester may have to build a model of the proposed system, perhaps by interviewing key stakeholders to understand what their expectations are. However formal or informal the model is, and however it is built, it provides a test basis from which we can generate tests systematically.
Remember, also, that the specification can contain non-functional elements as well as functions; topics such as reliability, usability and performance are examples. These need to be systematically tested as well.
What we need, then, are techniques that can explore the specified behaviour systematically and thoroughly in a way that is as efficient as we can make it. In addition, we use what we know about software to ‘home in‘ on problems; each of the test case design techniques is based on some simple principles that arise from what we know in general about software behaviour.
You need to know five specification-based techniques for the Foundation Certificate:
Equivalence partitioning
Boundary value analysis
Decision table testing
State transition testing
Use case testing
You should be capable of generating test cases for the first four of these techniques.
|
CHECK OF UNDERSTANDING
|
Equivalence partitioning
Input partitions
Equivalence partitioning is based on a very simple idea: it is that in many cases the inputs to a program can be ‘chunked‘ into groups of similar inputs. For example, a program that accepts integer values can accept as valid any input that is an integer (i.e. a whole number) and should reject anything else (such as a real number or a character). The range of integers is infinite, though the computer will limit this to some finite value in both the negative and positive directions (simply because it can only handle numbers of a certain size; it is a finite machine). Let us suppose, for the sake of an example, that the program accepts any value between ‘10,000 and +10,000 (computers actually represent numbers in binary form, which makes the numbers look much less like the ones we are familiar with, but we will stick to a familiar representation). If we imagine a program that separates numbers into two groups according to whether they are positive or negative the total range of integers could be split into three ‘partitions‘: the values that are less than zero; zero; and the values that are greater than zero. Each of these is known as an ‘equivalence partition‘ because every value inside the partition is exactly equivalent to any other value as far as our program is concerned. So if the computer accepts ‘2,905 as a valid negative integer we would expect it also to accept ‘3. Similarly, if it accepts 100 it should also accept 2,345 as a positive integer. Note that we are treating zero as a special case. We could, if we chose to, include zero with the positive integers, but my rudimentary specification did not specify that clearly, so it is really left as an undefined value (and it is not untypical to find such ambiguities or undefined areas in specifications). It often suits us to treat zero as a special case for testing where ranges of numbers are involved; we treat it as an equivalence partition with only one member. So we have three valid equivalence partitions in this case.
The equivalence partitioning technique takes advantage of the properties of equivalence partitions to reduce the number of test cases we need to write. Since all the values in an equivalence partition are handled in exactly the same way by a given program, we need only test one of them as a representative of the partition. In the example given, then, we need any positive integer, any negative integer and zero. We generally select values somewhere near the middle of each partition, so we might choose, say, −5,000, 0 and 5,000 as our representatives. These three test inputs would exercise all three partitions and the theory tells us that if the program treats these three values correctly it is very likely to treat all of the other values, all 19,998 of them in this case, correctly.
The partitions we have identified so far are called valid equivalence partitions because they partition the collection of valid inputs, but there are other possible inputs to this program that would not be valid—real numbers, for example. We also have two input partitions of integers that are not valid: integers less than —10,000 and integers greater than 10,000. We should test that the program does not accept these, which is just as important as the program accepting valid inputs.
Non-valid partitions are also important to test. If you think about the example we have been using you will soon recognise that there are far more possible non-valid inputs than valid ones, since all the real numbers (e.g. numbers containing decimals) and all characters are non-valid in this case. It is generally the case that there are far more ways to provide incorrect input than there are to provide correct input; as a result, we need to ensure that we have tested the program against the possible non-valid inputs. Here again equivalence partitioning comes to our aid: all real numbers are equally non-valid, as are all alphabetic characters. These represent two non-valid partitions that we should test, using values such as 9.45 and ‘r‘ respectively. There will be many other possible non-valid input partitions, so we may have to limit the test cases to the ones that are most likely to crop up in a real situation.
|
EXAMPLE EQUIVALENCE PARTITIONS
|
Exercise 4.1
| Exercise 4.1 |
Suppose you have a bank account that offers variable interest rates: 0.5 per cent for the first £1,000 credit; 1 per cent for the next £1,000; 1.5 per cent for the rest. If you wanted to check that the bank was handling your account correctly what valid input partitions might you use?
The answer can be found at the end of the chapter. |
Output partitions
Just as the input to a program can be partitioned, so can the output. The program in the exercise above could produce outputs of 0.5 per cent, 1 per cent and 1.5 per cent, so we could use test cases that generate each of these outputs as an alternative to generating input partitions. An input value in the range £0.00££1,000.00 would generate the 0.5 per cent output; a value in the range £1,001.00££2,000.00 would generate the 1 per cent output; a value greater than £2,000.00 would generate the 1.5 per cent output.
Other partitions
If we know enough about an application we may be able to partition other values instead of or as well as input and output. For example, if a program handles input requests by placing them on one of a number of queues we could, in principle, check that requests end up on the right queue. In this case a stream of inputs can be partitioned according to the queue we anticipate it will be placed into. This is more technical and difficult than input or output partitioning but it is an option that can be considered when appropriate.
|
PARTITIONS—EXAMPLE 4.1 A mail-order company charges £2.95 postage for deliveries if the package weighs less than 2 kg, £3.95 if the package weighs 2 kg or more but less than 5 kg, and £5 for packages weighing 5 kg or more. Generate a set of valid test cases using equivalence partitioning. The valid input partitions are: under 2 kg; 2 kg or over but less than 5 kg; and 5 kg or over. Input values could be 1 kg, 3.5 kg, 7.5 kg. These would produce expected results of £2.95, £3.95 and £5 respectively. In this case there are no non-valid inputs (unless the scales fail). |
Exercise 4.2
| Exercise 4.2 |
A mail-order company selling flower seeds charges £3.95 for postage and packing on all orders up to £20 value and £4.95 for orders above £20 value and up to £40 value. For orders above £40 value there is no charge for postage and packing.
If you were using equivalence partitioning to prepare test cases for the postage and packing charges what valid partitions would you define? What about non-valid partitions? The answer can be found at the end of the chapter. |
Boundary value analysis
One thing we know about the kinds of mistakes that programmers make is that errors tend to cluster around boundaries. For example, if a program should accept a sequence of numbers between 1 and 10, the most likely fault will be that values just outside this range are incorrectly accepted or that values just inside the range are incorrectly rejected. In the programming world these faults coincide with particular programming structures such as the number of times a program loop is executed or the exact point at which a loop should stop executing.
This works well with our equivalence partitioning idea because partitions must have boundaries. A partition of integers between 1 and 99, for instance, has a lowest value, 1, and a highest value, 99. These are called boundary values. Actually they are called valid boundary values because they are the boundaries on the inside of a valid partition. What about the values on the outside? Yes, they have boundaries too. So the boundary of the non-valid values at the lower end will be zero because it is the first value you come to when you step outside the partition at the bottom end. (You can also think of this as the highest value inside the non-valid partition of integers that are less than one, of course.) At the top end of the range we also have a non-valid boundary value, 100.
This is the boundary value technique, more or less. For most practical purposes the boundary value analysis technique needs to identify just two values at each boundary. For reasons that need not detain us here there is an alternative version of the technique that uses three values at each boundary. For this variant, which is the one documented in BS 7925-2, we include one more value at each boundary when we use boundary value analysis: the rule is that we use the boundary value itself and one value (as close as you can get) either side of the boundary.
So, in this case lower boundary values will be 0, 1, 2 and upper boundary values will be 98, 99, 100. What does ‘as close as we can get‘ mean? It means take the next value in sequence using the precision that has been applied to the partition. If the numbers are to a precision of 0.01, for example, the lower boundary values would be 0.99, 1.00, 1.01 and the upper boundary values would be 98.99, 99.00, 99.01.
When you come to take your exam you will find that the exam recognises that there are two possible approaches to boundary value analysis. For this reason any questions about boundary value analysis will clearly signal whether you are expected to identify 2 or 3 values at any boundary. You will find that this causes no problems, but there are examples below using both the 2 value and the 3 value approach, just to be on the safe side and ensure that you do not get taken by surprise in the exam.
The best way to consolidate the idea of boundaries is to look at some examples.
|
BOUNDARY VALUES—EXAMPLE 4.2
|
Exercise 4.3
| Exercise 4.3 | A system is designed to accept scores from independent markers who have marked the same examination script. Each script should have 5 individual marks, each of which is out of 20, and a total for the script. Two markers’ scores are compared and differences greater than three in any question score or 10 overall are flagged for further examination. |
Using equivalence partitioning and boundary value analysis identify the boundary values that you would explore for this scenario.
(In practice, some of the boundary values might actually be in other equivalence partitions, and we do not need to test them twice, so the total number of boundary values requiring testing might be less than you might expect.)
The answer can be found at the end of the chapter.
|
CHECK OF UNDERSTANDING
|
Decision table testing
Specifications often contain business rules to define the functions of the system and the conditions under which each function operates. Individual decisions are usually simple, but the overall effect of these logical conditions can become quite complex. As testers we need to be able to assure ourselves that every combination of these conditions that might occur has been tested, so we need to capture all the decisions in a way that enables us to explore their combinations. The mechanism usually used to capture the logical decisions is called a decision table.
A decision table lists all the input conditions that can occur and all the actions that can arise from them. These are structured into a table as rows, with the conditions at the top of the table and the possible actions at the bottom. Business rules, which involve combinations of conditions to produce some combination of actions, are arranged across the top. Each column therefore represents a single business rule (or just ‘rule‘) and shows how input conditions combine to produce actions. Thus each column represents a possible test case, since it identifies both inputs and expected outputs. This is shown schematically in the box below.
In reality the number of conditions and actions can be quite large, but usually the number of combinations producing specific actions is relatively small. For this reason we do not enter every possible combination of conditions into our decision table, but restrict it to those combinations that correspond to business rules—this is called a limited entry decision table to distinguish it from a decision table with all combinations of inputs identified. In this chapter we will always mean the limited entry kind when we refer to a decision table.
As usual, we use an example to clarify what we mean.
|
DECISION TABLE TESTING—EXAMPLE 4.3 A supermarket has a loyalty scheme that is offered to all customers. Loyalty cardholders enjoy the benefits of either additional discounts on all purchases (rule 3) or the acquisition of loyalty points (rule 4), which can be converted into vouchers for the supermarket or to equivalent points in schemes run by partners. Customers without a loyalty card receive an additional discount only if they spend more than £100 on any one visit to the store (rule 2), otherwise only the special offers offered to all customers apply (rule 1). |
From the decision table we can determine test cases by setting values for the conditions and determining the expected output, e.g. from rule 1 we could input a normal customer with a £50 transaction and check that no discount was applied. The same customer with a £150 transaction (rule 2) should attract a discount. Thus we can see that each column of the decision table represents a possible test case. |
|
CHECK OF UNDERSTANDING
|
Exercise 4.4
| Exercise 4.4 | A mutual insurance company has decided to float its shares on the stock exchange and is offering its members rewards for their past custom at the time of flotation. Anyone with a current policy will benefit provided it is a ‘with-profits‘ policy and they have held it since 2001. Those who meet these criteria can opt for either a cash payment or an allocation of shares in the new company; those who have held a qualifying policy for less than the required time will be eligible for a cash payment but not for shares. Here is a decision table reflecting those rules. |
| Rule 1 | Rule 2 | Rule 3 | Rule 4 | |
|---|---|---|---|---|
| Conditions: | ||||
| Current policy holder | Y | Y | Y | N |
| Policy holder since 2001 | N | Y | N | - |
| ‘With-profits‘ policy | Y | Y | N | - |
| Actions: | ||||
| Eligible for cash payment | Y | Y | N | N |
| Eligible for share allocations | N | Y | N | N |
What expected result would you expect to get for the following test case?
Billy Bunter is a current policy holder who has held a ‘with-profits‘ policy since 2003.
The answer can be found at the end of the chapter.
State transition testing
The previous technique, decision table testing, is particularly useful in systems where combinations of input conditions produce various actions. Now we consider a similar technique, but this time we are concerned with systems in which outputs are triggered by changes to the input conditions, or changes of ‘state‘; in other words, behaviour depends on current state and past state, and it is the transitions that trigger system behaviour. It will be no surprise to learn that this technique is known as state transition testing or that the main diagram used in the technique is called a state transition diagram.
Look at the box to see an example of a state transition diagram.
|
STATE TRANSITION DIAGRAMS A state transition diagram is a representation of the behaviour of a system. It is made up from just two symbols. The first is The second is |
which is the symbol for a state. A state is just what it says it is: the system is ‘static‘, in a stable condition from which it will only change if it is stimulated by an event of some kind. For example, a TV stays ‘on‘ unless you turn it ‘off‘; a multifunction watch tells the time unless you change mode.
which is the symbol for a transition, i.e. a change from one state to another. The state change will be triggered by an event (e.g. pressing a button or switching a switch). The transition will be labelled with the event that caused it and any action that arises from it. So we might have ‘mode button pressed‘ as an event and ‘presentation changes‘ as the action. Usually (but not necessarily) the start state will have a double arrowhead pointing to it. Often the start state is obvious anyway.If we have a state transition diagram representation of a system we can analyse the behaviour in terms of what happens when a transition occurs.
Transitions are caused by events and they may generate outputs and/or changes of state. An event is anything that acts as a trigger for a change; it could be an input to the system, or it could be something inside the system that changes for some reason, such as a database field being updated.
In some cases an event generates an output, in others the event changes the system’s internal state without generating an output, and in still others an event may cause an output and a change of state. What happens for each change is always deducible from the state transition diagram.
|
STATE TRANSITION DIAGRAM—EXAMPLE 4.4 A hill-walker’s watch has two modes: Time and Altimeter. In Time mode, pressing the Mode switch causes the watch to switch to Alt mode; pressing Mode again returns to Time mode. While the watch is in Alt mode the Set button has no effect. When the watch is in Time mode pressing the Set button transitions the watch into Set Hrs, from which the Hrs display can be incremented by pressing the Set button. If the Mode switch is pressed while the watch is in Set Hrs mode the watch transitions to Set Mins mode, in which pressing the Set button increments the Mins display. If the Mode button is pressed in this mode the watch transitions back to Time mode (Figure 4.1). Note that not all events have an effect in all states. Where an event does not have an effect on a given state it is usually omitted, but it can be shown as an arrow starting from the state and returning to the same state to indicate that no transition takes place; this is sometimes known as a ‘null‘ transition or an ‘invalid‘ transition. |
Rather than work out what happens for each event each time we want to initiate a test, we can take the intermediate step of creating what is known as a state table (ST). An ST records all the possible events and all the possible states; for each combination of event and state it shows the outcome in terms of the new state and any outputs that are generated.
The ST is the source from which we usually derive test cases. It makes sense to do it this way because the analysis of state transitions takes time and can be a source of errors; it is better to do this task once and then have a simple way of generating tests from it than to do it every time we want to generate a new test case.
Here is an example of what an ST looks like.
Figure 4.1 State transition diagram of the hill-walker’s watch
|
|
STATE TABLE—EXAMPLE 4.4 An ST has a row for each state in the state transition diagram and a column for every event. For a given row and column intersection we read off the state from the state transition diagram and note what effect (if any) each event has. If the event has no effect we label the table entry with a symbol that indicates that nothing happens; this is sometimes called a ‘null‘ transition or an ‘invalid‘ transition. If the event does have an effective label the table entry with the state to which the system transitions when the given event occurs; if there is also an output (there is sometimes but not always) the output is indicated in the same table entry separated from the new state by the ‘/‘ symbol. The example shown in Table 4.1 is the ST for Figure 4.1, which we drew in the previous box.
|
Once we have an ST it is a simple exercise to generate the test cases that we need to exercise the functionality by triggering state changes.
|
STATE TRANSITION TESTING—EXAMPLE 4.4 We generate test cases by stepping through the ST. If we begin in Time mode then the first test case might be to press Mode and observe that the watch changes to Alt state; pressing Mode again becomes test case 2, which returns the watch to Time state. Test case 3 could press Set and observe the change to Set Hrs mode and then try a number of presses of Set to check that the incrementing mechanism works. In this way we can work our way systematically round the ST until every single transition has been exercised. If we want to be more sophisticated we can exercise pairs of transitions, e.g. pressing Set twice as a single test case, to check that Hrs increments correctly. We should also test all the negative cases, i.e. those cases where the ST indicates there is no valid transition. |
|
CHECK OF UNDERSTANDING
|
Exercise 4.5
| Exercise 4.5 |
In the state transition diagram in Figure 4.2, which of the sequences of transitions below would be valid?
ABCDE FEABC ABCEF EFADC |
The answer can be found at the end of the chapter.
Figure 4.2 State transition diagram
Use case testing
Use cases are one way of specifying functionality as business scenarios or process flows. They capture the individual interactions between ‘actors‘ and the system. An actor represents a particular type of user and the use cases capture the interactions that each user takes part in to produce some output that is of value. Test cases based on use cases at the business process level, often called scenarios, are particularly useful in exercising business rules or process flows and will often identify gaps or weaknesses in these that would not be found by exercising individual components in isolation. This makes use case testing very effective in defining acceptance tests because the use cases represent actual likely use.
Use cases may also be defined at the system level, with preconditions that define the state the system needs to be in on entry to a use case to enable the use case to complete successfully, and postconditions that define the state of the system on completion of the use case. Use cases typically have a mainstream path, defining the expected behaviour, and one or more alternative paths related to such aspects as error conditions. Well defined use cases can therefore be an excellent basis for system level testing, and they can also help to uncover integration defects caused by incorrect interaction or communication between components.
In practice, writing a test case to represent each use case is often a good starting point for testing, and use case testing can be combined with other specification-based testing.
|
USE CASES In a use case diagram (e.g. Figure 4.3) each type of user is known as an actor, and an actor stands for all users of the type. Use cases are activities carried out for that actor by the system. This is, in effect, a high-level view of requirements. The diagram alone does not provide enough detail for testing, so we need some textual description of the processes involved as well. |
Use case testing has the major benefit that it relates to real user processes, so it offers an opportunity to exercise a complete process flow. The principles applied elsewhere can be applied here: first test the highest priority (highest value) use cases by taking typical examples; then exercise some attempts at incorrect process flows; and then exercise the boundaries.
Figure 4.3 Use case example
|
CHECK OF UNDERSTANDING
|