Software Testing: Test-Driven Development, Release Testing, and User Testing

8.1 Recap: The Three Stages of Program Testing

8.1.1 The Three Stages

Program testing happens in three broad stages, and it helps to keep them apart because each one has a different owner and a different goal:

1. Development testing — done by the developers themselves, during development.
2. Release testing — done by a separate testing team, which tests a complete version of the system — or a module — before it is released to its customers.
3. User testing — done by the users, or potential users, of the system, who test it in their own environment.

These three stages were discussed within the context of the testing process and typical plan-driven development processes.

A concrete picture to carry around. Think of a restaurant kitchen. The cook tastes the dish while cooking it — that is development testing, where the person who made the food checks it as they go. Before the dish leaves the kitchen, a separate person, maybe the head chef or a food-safety checker, inspects the finished plate and decides whether it is good enough to serve — that is release testing. Finally, the customer eats the dish at their own table and decides whether to come back — that is user testing. The cook, the checker, and the customer look for different things and trust different evidence. The analogy bends at one point: a customer's verdict is informal and continuous, while acceptance testing is a formal, one-time decision that may even be written into a contract.

The testing process in a plan-driven project. Testing is not one single action but a short loop of four steps: design test cases (a test case specifies the inputs, the expected output, and the statement of what is being tested), prepare test data (the actual input values used in the test run), run the program with the test data, and compare the results to the test cases. Designing test cases always needs people who understand what the system is supposed to do — the expected results cannot be invented by a machine. Execution, though, can be automated: the test program compares the system's actual output against the predicted output and reports any difference, with no human needing to eyeball the results.

The three stages are not alternatives to choose among — a serious project uses all three in sequence, because no single stage can cover the ground of the other two.

8.1.2 Development Testing: Unit, Component, and System

Development testing works at three levels, each testing a bigger slice of the system:

1. Unit testing tests individual objects and methods. It is typically automated, and test cases are chosen using partition testing (divide the possible inputs into groups that should be handled the same way, then pick test values from within each group) and testing guidelines.
2. Component testing tests related groups of objects. The focus is the interfaces between the components — the way data and calls cross from one component to another — because interface problems only show up when components interact.
3. System testing tests partial or completed systems. Beyond checking all the functionality, it must look at the emergent behavior of the system — the properties that only appear when everything works together, such as reliability, maintainability, and usability.

Emergent behavior (a property that exists only in the assembled whole) is the reason system testing exists. A unit can be flawless in isolation and the assembled system can still crash, corrupt data, or run too slowly, because the problems live in the interactions between the parts, not inside any single part.

For system testing you can develop tests from use cases, and you can use sequence diagrams along with the use cases to derive the tests. A sequence diagram shows the order in which objects send messages to one another, so it tells you which inputs to provide and which outputs to expect at each step of a thread. A variety of testing policies can be applied — for example, test every function that is reachable from a menu, test combinations of functions accessed through the same menu, or test every function with both correct and incorrect input.

8.1.3 Verification, Validation, Inspections, and Reviews

Before diving into execution-based program testing, the distinction between verification and validation was drawn, and the role of inspections and reviews in verification and validation was discussed. The key point carried forward: inspections and reviews are a verification and validation activity in their own right, complementary to running tests on the program.

Two one-line questions hold the whole distinction:

• Verification asks: are we building the product right? It checks that the software meets its stated functional and non-functional requirements — that the specification has been implemented correctly.
• Validation asks: are we building the right product? It checks that the software does what the customer actually expects, which goes beyond the written specification, because requirements do not always reflect the customer's real needs.

Inspections are careful, line-by-line examinations of the system requirements, design models, or source code by people who know the system and its application domain. They are static techniques: the software is never executed. Anything readable can be inspected — requirements documents, architecture models, database schemas, program code, even proposed system tests. Inspections bring three advantages that testing does not have: during execution one error can mask another, so you can never be sure whether a later anomaly is new or a side effect of the first error, while an inspection session can uncover many independent errors at once; incomplete versions of a system can be inspected without building special test harnesses; and an inspection can also judge broader quality attributes, such as compliance with standards, portability, and maintainability.

8.2 Test-Driven Development

8.2.1 The Core Idea and the Five-Stage Cycle

Test-driven development (TDD) is an approach to program development in which you interleave testing and code development. You develop the code incrementally, together with a set of tests for that increment, and you do not start working on the next increment until the code you have developed passes all of its tests. The practice was originally part of agile development methods, but it has since gained mainstream acceptance and is used in plan-driven processes as well.

The problem TDD solves. When tests are written after the code, it is easy to rush them, to test only the happy path, or to skip them entirely when the deadline looms. When the test comes first, it acts as a specification that the code must satisfy, and it stays behind to guard that specification forever after. TDD makes testing an unavoidable part of building, not an afterthought.

Inputs and outputs of one cycle. The inputs are a requirement (usually one small increment) and the existing test suite. The output is working code that satisfies the new test, an expanded test suite, and confidence that nothing already built has broken.

The typical test-driven development process has five stages:

1. Identify the increment. The new functionality required should normally be small — implementable in a few lines of code.
2. Write the test case for this function, implemented as an automated test together with all the previous test cases. Because it is automated, the test can be executed and will report whether it passed or failed.
3. Run the test along with the other tests that have already been implemented. Since the new functionality has not been implemented yet, the new test will fail. This failure is deliberate: it shows that the test adds something to the existing test set. A test that passes before the code exists tells you nothing — it may be exercising nothing at all.
4. Implement the functionality and rerun all the test cases. This may involve refactoring existing code to improve it — restructuring it without changing its behavior — and adding new code to what is already there.
5. Move on to the next increment. Once all the tests run successfully, you identify and implement the next set of requirements.

There is a built-in diagnostic here: if the test fails at stage four, you know the problem is with the current, newly added code — not with the rest of the system, because all the earlier tests still pass.

8.2.2 Tests First Force You to Understand the Requirements

Writing the test before the code has a hidden benefit: if you do not know enough to write the test, you do not understand the requirements, and you will not develop the required code. The test-first order makes this visible immediately.

The professor's example: if your computation involves division, you should also check that you are not dividing the numbers by zero. If you forget to write the test for this, the checking code will never be included in the program. The test list becomes a checklist of everything the code must handle, and nothing gets silently skipped.

8.2.3 Automated Testing, JUnit, and Regression Testing

An automated test environment is essential for test-driven development, because the code is developed in very small increments and you must run every earlier test every time you add functionality or refactor the program. JUnit, the widely used automated testing framework for Java program testing, is the classic example. The tests are embedded in a separate program that runs the tests and invokes the system being tested. With such a setup you can run hundreds of separate tests in a couple of seconds.

An automated test in such a framework has three parts: a setup part that initializes the system with the test case — the inputs and the expected outputs; a call part that invokes the object or method being tested; and an assertion part that compares the result of the call with the expected result. If the assertion holds, the test passes; if not, the test fails and the framework reports exactly which test failed.

Running the full set of previously passing tests after each change is called regression testing. It matters because the test set grows incrementally as the program is developed, so you can always rerun it to check that a change has not introduced new bugs. The name comes from the idea of regression: a behavior that used to work and stops working after a change is said to have regressed, and this testing guards against exactly that.

8.2.4 The Benefits of Test-Driven Development

• Code coverage. In principle, every code segment you write should have at least one associated test, so you can be confident that all of the code in the system has actually been executed. Test coverage is one of the few testing outcomes you can measure, and TDD builds it in from the start.
• Early defect discovery. The code is tested as it is written, so defects are discovered early in the development process — when the code being checked is still small and the fix is still cheap — instead of months later when no one remembers what the code was supposed to do.
• Cheaper regression testing. Regression testing means running test sets that have successfully executed after changes have been made to a system, to check that the latest changes have not introduced new bugs and that the new code interacts as expected with the existing code. Manual regression testing is expensive and sometimes impractical: you have to run thousands of test cases for the entire system, it takes a lot of effort and time, and it is easy to miss certain important tests. Automation reduces the cost dramatically, because the existing tests can be rerun quickly and cheaply. All existing tests must run successfully before any further functionality is added, so as a programmer you can be confident that the new functionality has not broken the existing code.
• Simpler debugging. When a test fails in test-driven development, the problem's location is usually easy to pin down: only the latest test has failed, and it targets a small, newly written function. You do not need sophisticated debugging tools to locate the problem — the defect can only be in the most recently written layer of code. Reports of test-driven development use suggest that automated debuggers are used very little, or not even necessary.
• Documentation. Writing test cases before development produces documentation as a side effect: the tests themselves act as a form of documentation that describes what the code should be doing. A new programmer who wants to know what withdraw should do can read the test cases instead of guessing from the code.

Test-driven development — also called test-first development — is one of the most value-adding practices in new software development, where functionality is implemented with new code or with components from standard libraries. Programmers who have adopted it find it a useful and productive way to develop software, and it is claimed to encourage better structuring of the program and improved code quality, because you understand the requirements as well as the test cases. Keep in mind: these are empirical observations, not claims verified through experiments. Careful studies of the quality benefits so far have not given a decisive verdict.

8.2.5 Where Test-Driven Development Struggles

• Legacy systems and large reused components. If you are reusing large code components or legacy systems, you need to write tests for those systems as a whole. You cannot easily decompose them into separately testable components or elements, so incremental test-driven development is impractical there.
• Multithreaded systems. Different threads may be interleaved at different times in different test runs, so the runs may produce different results. Test-driven development alone cannot validate such a system, so you still need a system testing process to check that the system meets the requirements of all its stakeholders. System testing also tests the emergent properties of the system, like performance and reliability, and checks that the system does not do things it should not do, like producing unwanted outputs. Testing tools can be extended to integrate some aspects of system testing with test-driven development, but the system test remains a separate responsibility.

8.2.6 Student Questions and Answers

The exchange carries a warning as well as encouragement: partial adoption still helps, but the benefit comes from the order — requirements must be made testable before the code exists, or the practice quietly loses its edge.

8.3 Release Testing

8.3.1 What Release Testing Is

Release testing is the process of testing a particular release of a system that is intended for use outside the development team. The testing is done by a team outside the development team, usually the internal customers of a release to a user, or an external customer. Normally a system release is for customers and users, but in a complex project the release could be for other teams that are developing related systems. For software products — like Adobe products or Microsoft products — the release could be for the product management teams, who then prepare it for marketing and sales.

Two aspects distinguish release testing from the system testing discussed earlier:

• Release testing is done by an independent team, not by the development team.
• It is a form of system testing, but with a small distinction in purpose. Release testing is a process of validation testing: it ensures that the system meets its requirements and is good enough for release to the system's customers. System testing by the development team, in contrast, focuses on discovering bugs in the system — it is defect testing.

The primary goal of release testing is to show that the system delivers specific functionality and performance, that it has desirable properties such as dependability and usability, and that it does not fail during normal use.

Why must the team be independent? A development team has spent months inside the system and knows its quirks, so its members may unconsciously test what they already believe works, and they may be tempted to explain away a failure they recognize. A separate team reads the specification cold, follows it literally, and reports what it finds. Independence of the team and validation of the requirements are what make release testing a different activity from the development team's own system testing.

8.3.2 Black Box and Functional Testing

Release testing is usually a black box testing process. The tests are derived from the system specification, and the system is treated as a black box whose behavior can only be determined by studying the inputs and the related outputs. Another name for release testing is functional testing, because the tester is concerned only with the functionality — with what the system does — not with the code or the implementation.

Because it is black box testing, the approach is based on the requirements specification alone, not on the code or the implementation. There are several ways to run release testing, and we look at each in turn.

A picture of the black box. Draw the system as a closed rectangle with an arrow in and an arrow out. All possible inputs the system might receive form an input set $I$, and all possible outputs form an output set $O$. A few special inputs — say, an unexpected combination of keystrokes — make the system behave wrongly; call those inputs $I_e$ (e for erroneous) and the wrong outputs they trigger $O_e$. Defect testing hunts for inputs inside $I_e$, because those reveal problems; validation testing feeds the system normal, expected inputs and checks that the outputs are the correct members of $O$. The two moods of testing sit in the same picture: one tries to break the system, the other watches it behave correctly. In release testing, the black box is probed from the outside, using only what the specification promises about inputs and outputs.

8.3.3 Requirements-Based Testing

A general principle of good requirements engineering practice — and one of the important criteria a requirement must satisfy — is that requirements should be testable. A requirement should be written so that a test can be designed for it, and so that a tester can check that the requirement has been satisfied. In practice this means you can write acceptance test cases for each requirement and check them during acceptance testing.

Requirements-based testing is a systematic approach to test case design: you consider each requirement and test that the system has implemented all its requirements as stated in the requirements specification. The running example is the Mental Health Care Patient Information System, seen earlier in the context of system models. One requirement it illustrates: if a patient is allergic to a particular medication, any prescription of that medication shall result in a warning message to the system user who is prescribing; and if the prescriber — the doctor — chooses to ignore the allergy warning, he should provide a justification for why the warning has been ignored.

Checking this one requirement takes several test cases:

The lesson from this list: testing a requirement does not mean writing a single test. You have to write several tests to ensure adequate coverage of the requirement. You should also keep traceability records of your requirements-based testing, linking each test to the specific requirement it tested — so that when a requirement changes, you can find every test that must be revisited.

8.3.4 Scenario-Based Testing

Scenario-based testing is an approach to release testing in which you use scenarios to develop test cases for the system. A scenario is a story that describes one way in which the system might be used. It should be realistic, and real system users should be able to relate to it. If you have used scenarios or user stories as part of the requirements engineering process, you may be able to reuse them as testing scenarios.

For the Mental Health Care system, one scenario could be a home visit. A single run of that scenario can test a number of features at once: authentication; downloading and uploading specified patient records; home visit scheduling; encryption and decryption of patient records on mobile devices; recording and retrieval of patient information; modification of records; and the links with the drugs database that maintains information about side effects, including the prompts when a warning is ignored.

As a release tester you should run through the scenario playing the role of the user, and you may make some deliberate mistakes — for example, inputting wrong input — to check how the system responds to errors. You should carefully note any problems that arise, including performance problems: if the system is too slow, that changes the way it is used. If encrypting a record takes too long, for instance, busy users may skip the step and leave patient data exposed on a lost laptop.

Because a scenario checks several requirements within the same run, you should also check that combinations of requirements do not cause problems during release testing — not just that each individual requirement works on its own. A feature can be perfect in isolation and still break when it runs together with the next feature in the story.

8.3.5 Performance Testing

Once the system has been completely integrated, it is possible to test emergent properties such as performance and reliability. Performance tests have to be designed to ensure that the system can process its intended load. This usually involves running a series of tests in which you increase the load until the system's performance becomes unacceptable.

To test whether performance requirements are really being met, you may have to build an operational profile — a set of tests that reflect the actual mix of work the system will handle in practice. If 90% of the transactions in a real deployment are of type A, 5% of type B, and the remainder are of types C, D, and E, then the tests must be mostly type A. Otherwise the test measures the wrong mix and does not tell you what performance the users will actually experience.

Performance testing is concerned both with showing that the system meets its requirements and with discovering problems and defects in the system. It is not the best approach for defect testing, but it has been shown to be an effective way to discover defects that test the limits of the system.

8.3.6 Stress Testing

Stress testing is done as part of performance testing. It makes demands that are outside the design limits of the software, and it helps you test the failure behavior of the system: how the system behaves when an unexpected load — over and above the maximum prescribed load — is placed on it.

The system should not crash or cause data corruption; it should fail gracefully rather than collapse under the load. Stress testing can also reveal defects that only show when the system is fully loaded — unusual combinations of circumstances that ordinary testing never produces. It is particularly relevant to distributed systems based on networks of processors, where it helps you discover when degradation begins, so that you can add checks to the system to reject transactions beyond a certain limit.

8.4 User Testing and Acceptance Testing

8.4.1 Types of User Testing

The third stage of testing in the software development lifecycle is user testing. In user testing, customers or system users provide the test data and check that the tests are successful. The types of user testing covered here include alpha testing, beta testing, and acceptance testing — and acceptance testing, the one that decides whether the software may be accepted at all, gets the most attention.

Three distinct arrangements:

• Alpha testing — a selected group of software users works closely with the development team to test early releases of the software, while it is still being developed. Users see new features early and can flag problems that the developers, working only from requirements, cannot anticipate; developers get realistic feedback that improves the design of later tests. Alpha testing is often used for software products and apps, and agile methods effectively rely on it when users sit inside the development team.
• Beta testing — a release of the software is made available to a larger group of users, who experiment with it and raise the problems they discover. Beta testing matters most for software products used in many different settings: the product developer cannot know or replicate every environment, so a large, varied group of users finds the interaction problems the lab never will. It is also a form of marketing — users learn what the product can do for them.
• Acceptance testing — customers test the system, using their own data, to decide whether it is ready to be accepted from the system developers and deployed in the customer environment. Acceptance is a business decision: for custom systems, acceptance typically means the final payment for the software is made.

When to pick which: use alpha and beta when the product will meet environments you cannot predict; use acceptance testing whenever a customer will formally accept — and pay for — the system.

8.4.2 Embedded Users and Acceptance Tests in Agile Development

In agile development, an embedded user — a user who sits inside the development team — should plan the tests for each and every functionality or feature that is to be implemented. That person is also responsible for defining the test cases for each increment and deciding whether the developed software supports the user stories and meets the requirements.

The tests developed as part of test-driven development are usually equivalent to acceptance tests. The test cases and the running of the tests are automated, and development does not proceed until the acceptance tests — the test-driven development suite — have been fully executed and pass, using regression testing.

When users are embedded in a software development team, they should ideally be typical users of the system, with general knowledge of how the system will be used in practice. However, it can be difficult to find such users who are available for a long time throughout the development period. As a result, the acceptance tests may not be a true reflection of how the system is actually used in practice.

Also, the requirement for automated testing limits the flexibility of testing interactive systems, which can only be observed — their behavior has to be seen and felt. For such systems, acceptance testing may require groups of end users to use the system as if it were part of their everyday work.

8.4.3 Combining Agile Testing with Traditional Acceptance Testing

While an embedded user is an attractive notion in principle, it does not lead to high quality tests of the system at the acceptance test level. The problem of user involvement in agile teams is one reason why many companies use a combination of agile testing and more traditional system testing and acceptance testing: the system may be developed using agile techniques, but after the completion of a major release, a separate acceptance testing effort is still used to decide whether the system should be accepted.

8.5 Conclusion: What Testing Can and Cannot Do

8.5.1 Presence of Errors, Not Absence of Faults

The most important takeaway of the two sessions on software testing: testing can only show the presence of errors in a program. It cannot prove that there are no remaining faults. Exhaustive testing is impossible — there is a combinatorial explosion in the number of possible test cases, and the time and effort involved in exhaustive testing may not even be beneficial.

Why is exhaustive testing impossible? A program has many possible inputs, and each input can arrive in many orders with many combinations of state. The number of possible test cases grows so fast — the combinatorial explosion — that running them all is not feasible for any real system, and even if it were, much of the effort would be wasted testing near-identical behavior. Testing is always a sampling job, then: a small, carefully chosen subset of all possible test cases.

8.5.2 The Three Stages at a Glance

• Development testing is the responsibility of the software development team. It includes unit testing (testing individual objects and methods), component testing (testing related groups of objects), and system testing (testing partial or completed systems).
• Release testing is the responsibility of a separate team, which tests the system before it is released to the customer.
• In user testing, customers or system users provide the test data and check that the tests are successful.

Program testing is often defect testing: you try to break the software, using experience and guidelines to choose the types of test cases that have been effective in discovering defects in similar kinds of systems. Wherever possible, you should automate the tests — the tests are embedded in a program that can be run every time a change is made to the system.

8.5.3 Test-First Development, Scenario Testing, and Acceptance Testing

Test-first development, covered today, is an approach to development in which tests are written before the code to be tested is developed. Small changes are made to the code, and the code is refactored until all the tests execute successfully. Scenario testing, in the case of release testing, replicates the practical use of the system: you invent a typical usage scenario and use it to derive test cases. Finally, acceptance testing is the user testing process in which the aim is to decide whether the software is good enough to be deployed and used in its planned operational environment.

Exam Guidance Summary

• Exam note: The syllabus for the mid-semester test will be posted. It covers the topics studied up to and including today's session — the first eight sessions — together with the corresponding chapters from the textbook and the courseware modules.
• Exam note: The mid-semester exam is scheduled for Saturday 23rd. It is a closed book test: no access to textbooks, reference books, slides, or other resources.
• Exam note: The answers will be conceptual, based on the topics covered primarily in class. Go through the slides uploaded for this session and the corresponding chapters in the textbook, and be prepared to explore the points to ponder — the questions given at the end of the chapters.

Key Industry Applications

• Real-world: JUnit is the standard automated test environment for Java program testing and the typical tool that makes test-driven development practical in Java projects. Its setup, call, and assertion structure lets a Java team run the whole test suite in seconds after every change, which is exactly the speed test-first development depends on.
• Real-world: product software vendors such as Adobe and Microsoft release to product management teams, which then prepare the release for marketing and sales. For such vendors the "customer" of release testing is internal — the product management team — and the release test must convince them that the product is fit to sell.
• Real-world: the Mental Health Care Patient Information System shows how a real requirement — the allergy warning with an override justification — drives multiple test cases in requirements-based release testing, and how a realistic scenario such as the home visit exercises many features — authentication, record transfer, encryption, and the drugs database — in one run.
• Real-world: stress testing matters most for distributed systems built on networks of processors, where you can add transaction limits once the start of degradation is detected. Online services use this pattern to protect themselves: when the backend begins to slow under load, new requests are rejected early rather than served badly.