Testing and Evaluation

Overview

Recall questions on demonstrating correctness, normal, boundary and erroneous test data, acceptance testing and its evidence, the criteria and limits of evaluation, and sequential against iterative development.

What you need to know

Answer each one before you reveal it.

Questions from this subtopic

Each one with the model answer underneath.

1. 1

   What should a developer be able to do to demonstrate that a program is correct and efficient?

   Model answer

   They should be able to explain how the program works and then justify two claims about it — that it gives the right answer and that it does so economically — drawing on three kinds of evidence. Logical reasoning is an argument from the algorithm itself that it must be right for every possible input, including those never tried, together with an argument about how its running time grows as the input grows. Test results are the recorded outcome of a planned set of tests spanning normal, boundary and invalid inputs. Feedback from the intended users is evidence that it does what they need under real conditions. Each covers a weakness in the others: reasoning reaches cases no test can, testing exposes flaws in the reasoning, and the users catch the program that is faultless and yet solves the wrong problem.

2. 2

   Why must an implementation be tested for the presence of errors?

   Model answer

   Because a program that compiles and runs is not thereby correct: it may produce wrong results, fail on inputs the developer did not anticipate, or crash on invalid data. Testing is a deliberate attempt to find those errors before the users do, using selected test data rather than whatever comes to hand. Note that testing can show the presence of errors but cannot prove there are none, since only a small fraction of possible inputs can ever be tried — which is why the test data has to be chosen to cover the different kinds of case.

3. 3

   Name and explain the three types of test data that selected test data should cover.

   Model answer

   Normal (typical) data — values well inside the range the system is meant to accept, which the program should process correctly in the ordinary way. Boundary data — values at the very edges of the acceptable range and immediately outside it, because errors most often occur where the limits are handled, for example a comparison written as less-than where it should be less-than-or-equal. Erroneous data — values the system should reject, such as the wrong data type, an empty entry or a value far outside the range, which the program should refuse gracefully with a helpful message rather than crashing or accepting them.

4. 4

   A program accepts an examination mark from 0 to 100 inclusive. Give suitable normal, boundary and erroneous test data.

   Model answer

   Normal: 45, or any value comfortably inside the range such as 72. Boundary: 0 and 100, which must be accepted, together with -1 and 101, which must be rejected; testing both the last valid value and the first invalid value on each edge is what catches an incorrect comparison. Erroneous: a non-numeric entry such as 'fifty', an empty entry, a non-integer such as 63.5 if only whole marks are allowed, and a wildly out-of-range value such as 1000. Each test needs an expected result recorded before it is run, so that the actual result can be judged against it.

5. 5

   What is acceptance testing, and who carries it out?

   Model answer

   Acceptance testing is testing carried out with the intended users of the system, using it in the way they will actually use it, to confirm that the solution delivered meets what was agreed at the start. It is done by the users rather than by the developer. Its purpose is different from the developer's own testing: the developer tests whether the program does what the developer intended, while acceptance testing asks whether what was intended is what the user needed, and whether the system works in the real setting with real data and real working practices.

6. 6

   Why can a system pass all the developer's tests and still fail acceptance testing?

   Model answer

   Because the developer's tests check the program against the developer's understanding of the requirements. If that understanding was wrong, the tests are wrong in the same way and will all pass. Acceptance testing brings in the users' knowledge of the actual task, so it can reveal a missing requirement, a workflow that does not match how the job is really done, data that occurs in practice but was never anticipated, or performance that is acceptable on a test file and unacceptable on a full one.

Other ways to work this

The same content, four other ways.

What is inside Levo

Six ways to work the same material.