Software Testing

Program testing is the activity that catches problems in software before anyone trusts it: you run the program on artificial data and inspect what comes back. This set of notes covers what program testing is and its two goals, verification and validation, inspections versus execution-based testing, the testing process with its three stages (development, release, and user testing), and two techniques for choosing test cases — partition testing and guideline-based testing. It ends with test-first development, where you design the tests before you write the code and run them automatically.

Across these notes, one question keeps returning: how do you know that a program is good enough to trust? Testing gives a practical, evidence-based answer — but, as you will see, an incomplete one. Section 7.1 defines program testing and its two goals (validation testing and defect testing) and introduces the black-box model of inputs and outputs. Section 7.2 places testing inside the wider verification and validation process, with Barry Boehm's classic distinction between building the product right and building the right product. Section 7.3 contrasts execution-based testing with static inspections and their three advantages. Section 7.4 lays out the testing process — test cases, test data, the three stages of testing, and the limits of automation. Sections 7.5 to 7.9 go from unit testing through partition testing, guideline-based testing, component testing, and system testing. Section 7.10 closes with test-driven development: tests written before the code, run automatically, and rerun after every change.

7.1 Program Testing and Its Two Goals

7.1.1 What Program Testing Is

Program testing is intended to do two things at once: show that a program does what it is intended to do, and discover program defects before the program or the product is put into use or delivered. When you test software, you execute the program using artificial data — data chosen for the test, not real usage data. Then you check the results of the test run for errors, anomalies, or information about the program's non-functional attributes, such as how fast it runs or how usable it is.

Why artificial data? Real usage data is messy, arrives late, and cannot be aimed. Test data is designed: you pick inputs that target one requirement or one suspected weak spot, so each test run returns a precise answer about a specific part of the program. A defect (also called a bug) is a flaw in the program that makes it behave incorrectly — a crash, an unwanted interaction with another system, an incorrect computation, or corrupted data.

Testing is also a demonstration, for the developer and for the customer, that the software meets its requirements. What that demonstration must cover depends on the kind of software:

• For custom-built software, built for specific customers, there should be at least one test for every requirement in the requirements document, showing that the system works as expected.
• For generic software products, there should be at least one test for all of the system features that will be included in the product release.
• You may also test combinations of features, to check for any unwanted interactions between those features.

7.1.2 The Two Goals: Validation Testing and Defect Testing

Testing has two goals, and they pull in opposite directions.

Validation testing — show that the software works as expected. You use a set of test cases that reflect the system's expected use, and a successful test is one where the system operates as intended.

Defect testing — find as many defects as possible before the product is released. The test cases can be deliberately obscure and need not reflect how the system is normally used. A successful defect test is a test case that makes the system perform incorrectly — it exposes a defect.

The boundary between the two approaches is a little blurred in practice: during validation testing you will find defects in the system, and during defect testing many tests will show that the system works as per its requirements. The real difference is the goal. Validation testing shows to the customer and the developer that the software meets its requirements. Defect testing discovers faults where the behavior is incorrect or not in conformance with the specification. Consider division by zero: the system should throw an error, because the operation is not valid, and a test that provokes that error is a defect-revealing test of the system's error handling.

When to pick which: every system needs both — validation gives the customer confidence, defect testing gives the developers the bug list they must fix before that confidence is earned.

7.1.3 The Black-Box View: Inputs, Outputs, and the Erroneous Region

Think of the program or system being tested as a black box: you cannot see inside, only feed it inputs and observe outputs. The system accepts inputs from the input set $I$ and generates outputs in the output set $O$. Some of the outputs will be erroneous — in the diagram these form a small shaded ellipse labelled $O_e$, the erroneous-output region.

Correspondingly, inside the input ellipse there is a shaded erroneous-input region $I_e$ — the set of inputs that drive the system into anomalous behavior. Defect testing prioritizes finding those inputs in $I_e$, because these are the ones that reveal problems with the system and show errors. Validation testing works with the correct inputs that lie outside the shaded ellipse: valid inputs produce valid outputs in $O$, showing that the system performs correctly and every requirement is met.

$$I_e\subset I\text{and}{O}_e\subset O$$

Visual intuition: picture two large ellipses side by side, labelled $I$ (left) and $O$ (right), with an arrow from $I$ to $O$ passing through a small box labelled "System". Inside the left ellipse, a small shaded ellipse $I_e$ marks the inputs that cause anomalous behavior; inside the right ellipse, a matching small shaded ellipse $O_e$ marks the outputs that reveal defects. The takeaway: the shaded regions are the enemy — test cases that land in $I_e$ expose bugs, while test cases that stay in the white part of $I$ demonstrate correct behavior.

7.1.4 Testing Shows Presence, Not Absence

Testing by itself cannot show that the software is completely free of defects, or that it will behave as expected in all circumstances. There is always the possibility that a test you overlooked could discover further problems with the system. Dijkstra, one of the earliest contributors to software engineering, stated this eloquently: testing can only show the presence of errors, not their absence. You can discover bugs through testing, but you cannot test and show that the software is 100% defect free.

Why is exhaustive testing impossible? Because the input space is astronomically large. Even a modest program with 10 inputs, each taking only 10 possible values, has $10^{10}$ (10 billion) input combinations — far too many to run. In practice you test a small subset of the possible test cases and argue from that evidence.

Real-world connection: In safety-critical industries — aviation autopilots, medical infusion pumps, railway signalling — the erroneous-input region $I_e$ is mapped with extreme care, because a defect that only appears on unusual inputs can cost lives. Regulators (for example, the authorities certifying flight software) effectively demand that defect testing demonstrate exactly how large $I_e$ is and what happens when inputs land inside it, not just that ordinary inputs work.

7.2 Verification and Validation

7.2.1 Verification Versus Validation

Testing is a part of the general verification and validation (V&V) process. You saw this early in the process models: in the V process model, all levels of testing — module/unit testing, subsystem integration testing, system testing, and acceptance testing — are planned along with the corresponding stages of requirements, high-level design, and detailed design. Each development stage on the left leg of the V has a matching test stage on the right leg, and a test level is designed from the same information as its matching development stage.

V&V also includes static verification techniques, which apply to documents or work products that are not executable programs: requirements documents, designs, code that is complete or incomplete, and other project documents. Reviews and inspections are the standard static techniques.

Barry Boehm, one of the earliest contributors to software engineering, made the classic distinction between the two words:

• Verification — the software should conform to its specification. You constantly check whether the software is built as per the requirement specification and the design. In one phrase: are we building the product right?
• Validation — the software does what the user really requires, meaning its real requirements, not what is stated in the requirement specification. In one phrase: are we building the right product?

Keep the two assignments straight: "conforms to its specification" belongs to verification; "does what the user really requires" belongs to validation.

Verification and validation are almost always used together. The combined processes check that the software being developed meets its specification and delivers the functionality expected by the customer. They start as soon as the requirements become available and continue through all the stages of the development process, as the V model shows.

Software verification is the process of checking that the software meets its stated functional as well as non-functional requirements. Validation is the more general process: its aim is to ensure that the software meets the customer's expectations. It goes beyond checking conformance with the specification, and shows that the software does what the customer expects it to do.

When to pick which: do both — verification keeps you honest against the documented specification, validation keeps you honest against the world the customer actually lives in.

7.2.2 Why Validation Matters

Validation is very important because the stated requirements may not reflect the real world. Statements in the requirements specification may be incomplete, contradictory, conflicting, or ambiguous. The stated requirements may simply not match the real requirements of the customers.

Think of the customers' real wishes as the target, and the requirements document as a sketch of that target drawn from interviews and assumptions. The sketch can miss details (incompleteness), argue with itself (contradictions and conflicts), and be readable in more than one way (ambiguity). If you only ever verify — check the product against the sketch — you can build a perfect product that solves the wrong problem. Validation is the process that checks the sketch itself against the target.

7.2.3 Confidence: Fit for Purpose

The goal of the verification and validation processes together is to establish confidence that the system is fit for its purpose — that it is good enough for its intended use. The level of required confidence depends on three factors.

Software purpose. The more critical the software, the more important that it be reliable, safe, and secure. The level of confidence required for software that controls a safety-critical system is much higher than that required for a prototype application of a new product idea. A prototype only needs to show that certain aspects can be tried out.

User expectations. Users often have low expectations of software quality because of prior experience with buggy and unreliable software; they are not surprised when the software fails. When a new system is installed, users may tolerate some failures, because the benefits of using the software can outweigh the costs of failure recovery. But as a product becomes established and is used on a regular basis, users expect it to be reliable — so later versions of the system, used regularly, may require more thorough testing.

Marketing environment. When a software company brings a system to market, it must take into account the competing products, the price that customers are willing to pay, and the required schedule for delivering the system. In a competitive environment, the company may decide to release a program before it has been fully tested and debugged, because it wants to be the first to enter the market. If the product is very cheap, users may be willing to tolerate a lower level of reliability — they are getting it for use, and they can even act as beta testers.

Real-world connection: The distinction pays off most visibly in regulated industries: medical device and aviation projects maintain separate verification (traceability from requirement to test) and validation (clinical or flight trials showing the device works for real users) evidence, because a certifying body will reject a system that is thoroughly verified against a spec that never matched the actual use case.

7.3 Inspections and Program Testing

7.3.1 Static Verification: Inspections, Reviews, Walkthroughs

In addition to software testing, the V&V process may involve software inspections — static verification techniques concerned with analyzing and checking the various documents of the software development lifecycle: system requirements, design models, program source code, test cases, and so on. No software needs to be executed for an inspection.

Dynamic verification, in contrast, is done only by executing a program and observing its behavior. This can be done only with executable programs. Even with incomplete programs you need other supporting software — test harnesses, and the calling and called modules — to test the parts that exist.

Inspections can be applied to a wide variety of work products at any stage of the lifecycle: the requirements specification document, starting with the earliest requirements elaboration stages; the high-level design (software architecture); the detailed design models; database schemas; code; test cases; and many other documents. Executable programs and executable prototypes get program testing instead.

When you inspect a system, people examine the artifacts using their knowledge of the system, its application domain, and the programming or modeling language. In that way they can discover a variety of errors — syntactic errors, semantic errors, omissions, ambiguities, and more.

7.3.2 Three Advantages of Inspections Over Testing

Inspections have three advantages over execution-based testing.

Error masking. During program testing, a single error can mask or hide other errors. If an exception is thrown, it may mask all the other errors, and you have to debug it first. When an error leads to unexpected outputs, you can never be sure whether later anomalies are new errors or side effects of the original error. Inspections are static techniques that execute nothing, so you never have to worry about interactions between errors: a single inspection session can discover many kinds of errors in the work products, including the programs.

Incomplete work products. You can inspect incomplete versions of a document without extra cost. If a program is incomplete, you need to develop specialized test harnesses to test the parts that are available, which adds to the system development costs.

Broader quality attributes. Apart from looking for defects, inspections can consider broader quality attributes of the program: compliance with programming standards, portability, maintainability, inefficiencies, inappropriate algorithms, and poor programming style that could make the system difficult to maintain and update.

7.3.3 What Inspections Cannot Do

Inspections cannot replace execution-based testing. They are not good for discovering defects that arise because of unexpected interactions between different parts of the program, timing problems, or problems with the system performance. In small companies or small development teams, it can also be difficult and expensive to put together a separate team for inspection, because all potential team members may also be the developers of the software.

As a complement, static analysis can analyze the source code of a program automatically using tools, and discover anomalies without executing anything. Inspections themselves are discussed in more detail later, when we discuss processes for quality management.

7.3.4 Student Questions and Answers

Real-world connection: Formal inspection is a standard practice in organizations that ship safety-critical or high-assurance software: structured Fagan-style inspections are estimated to catch well over half of the errors in a program before it is ever executed, which is why code review tools and pull-request review processes in modern development teams are the industry descendants of this idea.

7.4 The Testing Process

7.4.1 Test Cases and Test Data

Two terms sit at the heart of testing.

A test case is a specification of the inputs to the test plus the expected output — the expected test results. A complete test case has three parts: the test condition being tested, the set of inputs to be given to that particular test, and the set of expected outputs for each test input.

Test data are the inputs that have been derived from the test cases, designed to test the system.

Sometimes test data can be generated automatically based on the test case conditions — for example, "a set of integers with four digits". But automatic test case generation is impossible, because you must be able not only to specify the input condition but also to specify the expected output for the given code. The people who understand the system are the ones who specify the expected test results.

Test execution, in contrast, can be automated: the test results are automatically compared with the expected results, with no manual intervention to look for errors and anomalies. The system running the test cases generates test reports that list the errors and anomalies found — both valid and invalid outputs appear in the reports.

7.4.2 The Testing Process Model

In plan-driven development, the program testing process is a simple pipeline:

1. Design test cases. The work product is the test cases themselves.
2. Use the test cases to prepare test data.
3. Run the program with the test data, generating test results.
4. Compare the test results with the expected results in the test cases.
5. Generate a test report of the errors and anomalies — both valid and invalid outputs are reported.

7.4.3 The Three Stages of Testing

There are three stages of testing executable programs.

Development testing — testing during the software development process. A lot of testing, especially module code and unit testing, is done during the development phase of the software development lifecycle, whose four main phases are specification, development, validation, and evolution. The aim is to discover as many bugs and defects as possible before the program is handed over for release. The programmers and designers are likely to be involved.

Release testing — testing the complete version of the release, the system or the module, before it is released to users. The aim is to check that the system meets the requirements of the system stakeholders, even when the release is internal to another team.

User testing — actual or potential users of the system test it in their own environment. For software products, the "user" may be an internal marketing group that decides whether the software can be marketed, released, and sold. Acceptance testing is one type of user testing: the customer formally tests the system to decide whether it should be accepted from the developer, or whether further modifications or development are required.

7.4.4 Manual and Automated Testing

In practice, the testing process involves a mix of manual and automated testing. In manual testing, a tester runs the program with some test data and compares the results to their expectations, then notes and reports discrepancies to the program developers. In automated testing, the tests are encoded into a program that is run each time the system under development is to be tested. Automated testing is much faster than manual testing, especially for regression testing — rerunning all the previous tests to check that changes to the program have not introduced any new bugs.

But testing can never be completely automated. Automated tests can only check that a program does what it is supposed to do. It is practically impossible to automate testing of systems that depend on how things look and feel, like a graphical user interface, or to check that the program will not result in unanticipated side effects when various modules interact randomly at runtime.

Even so, test automation does improve productivity in software testing to a great extent. Many professionals use tools like JUnit or other frameworks for Java program testing, and many other testing frameworks, to run test cases automatically during regression testing. Test automation itself is covered in detail as a separate topic later.

Real-world connection: Continuous integration servers in modern software teams run exactly this pipeline on every code change: the automated part re-runs thousands of test cases and compares results in minutes (regression testing), while the human part — designing the test cases and the expected outputs — still decides what is worth checking. The division of labour is the same one the pipeline model draws.

7.5 Development Testing and Unit Testing

7.5.1 Development Testing

Development testing includes all testing activities carried out by the team developing the system. The tester is usually the programmer who develops the software. Some development processes use programmer-tester pairs — pair programming in extreme programming, where a programmer and a team member together develop and test the programs they develop. For safety-critical systems, you may have a separate testing group within the development team, responsible for developing tests and maintaining detailed records of the test results.

Within development testing there are three stages:

• Unit testing — individual programs, program units, or object classes are tested. Unit testing should focus on testing the functionality of objects or methods.
• Component testing — several individual units are integrated to create composite components. Component testing should focus on testing the component interfaces that provide access to the component functions.
• System testing — some or all of the components in the system are integrated and the system is tested as a whole. System testing should focus on testing the interactions between the components.

Development testing is primarily a defect testing process: you try to discover as many bugs or defects as possible. It is usually interleaved with debugging, which involves locating the problems in the code and correcting the programs to fix the bugs.

7.5.2 Unit Testing

Unit testing is the process of testing individual program components, such as methods or object classes. Individual functions or methods are the simplest types of components: your tests call these routines with different parameters, using various approaches to test case design.

When you test object classes, you should design your tests to provide coverage of all the features of the object: test all operations associated with the object; set and check the value of all attributes; and put the object into all possible states, which means simulating all events that cause a state change.

Inheritance in an object-oriented design makes object class testing a little complicated. You cannot simply test an operation in the class where it is defined and assume that it will work as expected in all of the subclasses that inherit the operation. The inherited operation may make assumptions about other operations and attributes, and those assumptions may not be valid in some subclasses. So you have to test the inherited operation everywhere it is used.

Automate unit testing wherever possible, using a test automation framework. Unit testing frameworks provide generic test classes that you extend to create specific test cases; the framework then runs all the tests you have implemented and reports, through a graphical user interface, on the success or otherwise of each test case.

Using a test framework, an entire test suite can often be run in a few seconds, so it is possible to execute all the tests every time you make a change to the program, as we do in regression testing.

A typical automated test has three parts. The setup part initializes the system with the test case — namely the inputs and the expected outputs. The calling part calls the object or method to be tested. The assertion part compares the result of the call with the expected test result: if the assertion evaluates true, the test is successful; otherwise the test has failed.

Sometimes the object you are testing has dependencies on other objects that may or may not have been implemented, and their use might slow down the testing process. If the object calls a database, for example, a slow setup process may be needed before it can be used. In such cases you may decide to use mock objects instead. A mock object has the same interface as the real dependency but simulates its functionality — a mock database holds a few data items in a fast array instead of touching disks, and a mock clock returns the times you need regardless of the real time.

7.5.3 Selecting Effective Unit Test Cases

As a programmer, how do you decide on selecting unit test cases? You choose effective unit test cases, which means two kinds of tests. One kind shows that the system does what it is supposed to do — the normal operation of the system, which is validation testing. The other kind shows that there are defects in the component, which should be revealed by the test cases — defect testing.

Two strategies help you choose test cases at the unit level:

• Partition testing — you identify groups of inputs that have certain common characteristics and are expected to be processed in the same way, then choose tests from within each of these groups.
• Guideline-based testing — you use testing guidelines to choose test cases; the guidelines might reflect previous experience of the kinds of errors that programmers often make when developing components.

Real-world connection: Modern unit testing practice rests on this exact toolkit: JUnit (Java), PyTest (Python), and Jest (JavaScript) provide the three-part automated test scaffolding, and mock objects (mock databases, mock clocks, mock network clients) are standard practice in every codebase that must test fast, deterministically, and before its dependencies exist.

7.6 Partition Testing and Equivalence Partitions

7.6.1 Equivalence Partitions

Partition testing rests on a simple observation: the input data and the output results of a program can be thought of as members of sets, or groups, with common characteristics. Sets of positive numbers, sets of negative numbers, sets of menu selections — programs normally behave in a comparable way for all members of a set. A program that takes a two-digit integer input will work the same way for all two-digit integers. If a program does a computation that requires two positive numbers, you would expect it to behave the same way for all positive numbers.

Because members of these sets behave equivalently, the sets are sometimes called equivalence partitions or equivalence domains.

The systematic approach to test case design is to identify all input partitions and output partitions for the system or component, then design test cases so that the inputs or outputs lie within these partitions. Partition testing can be used to design test cases for both systems and components, but it is used primarily for components or program-level unit testing.

Visual intuition: the large shaded ellipse on the left represents the set of all possible inputs to the program being tested, and the smaller white ellipses inside it are the equivalence partitions — sets of data that will result in equivalent behavior. Any member of one such set produces the same output behavior from the program: the program being tested should process all members of an input equivalence partition in the same way. On the output side, output equivalence partitions are partitions in which all of the outputs have something in common. Sometimes there is a one-to-one mapping between an input equivalence partition and a corresponding output equivalence partition, but this is not always the case: you may need to define a separate input equivalence partition whose only common characteristic is that its members generate outputs within the same output partition. The shaded area in the left ellipse represents the invalid inputs, and the shaded area in the right ellipse represents the exceptions that may occur in response to invalid inputs.

7.6.2 Boundaries and Midpoints

Once you have identified a set of equivalence partitions, you choose test cases from each of these partitions. A good rule of thumb for test case selection is to choose test cases on the boundaries of the partitions, plus test cases closer to the midpoint of the partition.

Why? Both designers and programmers tend to consider typical values of inputs when developing a system, and you test those by choosing the midpoint of the partition. Boundary values are often atypical — 0 may behave differently from other non-negative numbers — and so they are sometimes overlooked by developers. Program failures often occur when processing these atypical values.

You identify partitions by using the program specification or the user documentation, and from experience, where you predict the classes of input that are likely to detect errors.

7.6.3 Worked Example: 4–10 Inputs, 5-Digit Integers

7.6.4 Black Box and White Box Testing

Using the specification of a system to identify equivalence partitions is called black box testing: you need no knowledge of the program, the algorithm, or how the system works.

You may also do white box testing: you look inside the code and find other possible test cases — check whether all paths are handled, all branches are handled, and whether all exceptions are handled to provide valid outputs, and identify the ranges where some exception handling should be applied. White box testing is covered in a little more detail separately.

Equivalence partitioning is an effective approach to testing because it helps account for the errors that programmers often make when processing inputs at the edges of partitions.

Real-world connection: Equivalence partitioning is the workhorse of form and input validation testing in real products — date fields, age ranges, credit-card lengths, password rules. Test engineers routinely build exactly the table above for every numeric field in a specification, and the boundary values are where real products historically fail (dates on the 31st, ages at 17/18, prices at ₹99.99/₹100), because the boundary is where the developer's < versus <= mistakes live.

7.7 Guideline-Based Testing

7.7.1 Guidelines for Sequences, Arrays, and Lists

Test guidelines encapsulate knowledge of what kinds of test cases are effective for discovering errors. When you are testing programs with sequences, arrays, or lists, these guidelines help reveal defects:

1. Test the software with sequences of input that have only a single value. Programmers naturally think of sequences as made up of several values, and sometimes they embed that assumption in their programs. So a program presented with a single-value sequence may not work properly.
2. Use different sequences of different sizes in different tests. This decreases the chances that a program with defects will accidentally produce a correct output because of some accidental characteristic of the input.
3. Derive tests so that the first value, the middle value, and the last value of the sequence are accessed. This approach might reveal problems at the partition boundaries, since you check the boundary values and the midpoint values.
4. Test sequences with zero length. Programmers might assume a set of values for any input, and the program should fail when there are sequences of zero length.

7.7.2 General Guidelines

Some general guidelines apply beyond sequences:

• Choose inputs that force the system to generate all error messages.
• Design inputs that cause input buffers to overflow.
• Automatically repeat the same input or series of inputs many times, to see if the system behaves anomalously for the same input.
• Provide invalid inputs to force invalid outputs to be generated.
• Create exceptions by forcing certain computation results to be too large or too small.

7.7.3 Experience

As you gain experience with testing, you can develop your own guidelines about how to choose effective test cases.

Real-world connection: Buffer-overflow and extreme-result guidelines are not academic: forcing inputs to overflow fixed-size buffers and pushing computations past representable limits (integer overflow, too-large file sizes) are standard practice in security testing, because attackers feed exactly the inputs the guidelines prescribe to break products.

7.8 Component Testing

7.8.1 Components and Interface Types

Software components are made up of several interacting objects, so you can assume that the unit tests on the individual objects within the components have been completed when you start component testing. The focus of component testing is the component's interface: the boundary across which other components reach its functions.

There are different types of interfaces between program components, and so different types of interface errors can occur. Parameter interfaces pass data, or sometimes function references, from one component to another; the methods in an object have a parameter interface. Procedural interfaces are where a component encapsulates a set of procedures and is called by other components. There are also message passing interfaces, which you test as well. A fourth type completes the picture: shared memory interfaces, where one subsystem writes data into a block of memory that other subsystems read — common in embedded systems.

Interface errors fall into three classes: interface misuse — a calling component passes the wrong type, wrong order, or wrong number of parameters; interface misunderstanding — a calling component assumes behavior the called component does not provide (for example, calling a binary search with an unordered array); and timing errors — the producer and consumer of shared data operate at different speeds, so the consumer reads out-of-date information. For parameter interfaces you can check pointer parameters with null pointers, and design tests that cause a component to fail.

7.8.2 Interface Testing Guidelines

Several guidelines help you test interfaces:

• Examine the code to be tested and identify each call to an external component.
• Design a set of tests in which the values of the parameters to the external components are at the extreme ends of their ranges. These extreme values are the most likely to reveal interface inconsistencies.
• When you are passing pointers across an interface, test the interface with null pointer parameters, to see if it fails.
• When a component is called through a procedure interface, design tests that deliberately cause the component to fail. Differing failure assumptions are one of the most common problems, or misunderstandings, in the interface specifications.
• Use stress testing in message passing systems: design tests that generate many more messages than are likely to occur in practice. This can reveal timing problems.
• When several components interact through a shared memory, design tests that vary the order in which the components are activated. These tests may reveal implicit assumptions the programmer made about the order in which the shared data is produced and consumed.

Real-world: stress testing message-passing components with far more messages than expected in practice is a standard technique for finding timing problems in distributed systems.

7.8.3 Inspections for Interfaces

For interface testing, it is sometimes better to use inspections and reviews than execution-based program testing, because a single exception can mask all the other errors in the program. Inspections can concentrate on the component interfaces, and they can question the assumptions behind the interface behavior during the inspection process.

Real-world connection: The interface guidelines map directly onto distributed-systems practice: API contract tests check parameter shapes and extreme values, chaos and stress testing flood message queues to reveal timing defects, and null-pointer tests are standard for every public API boundary. Microservice architectures are effectively one large component test in production, which is why these same guidelines appear in every integration testing playbook.

7.9 System Testing

7.9.1 What System Testing Checks

System testing is done during development, at the developer's end — it is not user testing or acceptance testing, which happen at the user's side. After integrating all the components to create a version of the system, you test the integrated system as a whole.

System testing checks that the components are compatible, that they interact correctly, and that they transfer the right data at the right time across the interfaces. This closely overlaps with component testing, but there are two important differences.

First, during system testing, reusable components that have been separately developed, and other commercially available off-the-shelf systems that are bought, can be integrated with the newly developed components, and then the complete system is tested. Second, components developed by different team members or other teams may be integrated at this stage.

System testing is a collective effort rather than an individual process, whereas unit testing and component testing are individual efforts. In most companies, system testing might involve a separate independent testing team with no involvement from the designers or programmers: the team is given the specification and is expected to test the system independently.

7.9.2 Emergent Behavior and Interaction Testing

During system testing you are checking the emergent behavior of the system: behaviors that emerge only when the system is executed — when all the components are assembled and the system runs. Performance, usability, reliability, maintainability, and safety are all emergent behaviors; these are non-functional properties.

Some emergent behavior is planned and has to be tested. For example, you may integrate an authentication component with a component that updates a system database, and you then have a system feature that restricts information updating to only authorized users. Sometimes, however, the emergent behavior is unplanned: the system behaves in some anomalous way, and you may have to develop tests that check that the system is doing only what it is supposed to do.

System testing should focus on the interactions between the components and the objects that make up the system. You may also test reusable components or systems, to check that they work as expected when integrated with new components. This interaction testing should discover component bugs that are only revealed when a component is used by other components in the system, and it also helps find misunderstandings that developers may have about other components in the system.

7.9.3 Use-Case-Based Testing

Because of its focus on interactions, use case based testing is an effective approach to system testing. Several components or objects are normally implemented in each use case of a system, so testing the use case will force these interactions to occur. If you have developed a sequence diagram to model the use case implementation, you can see exactly which objects or components are involved in the interaction. For example, the interactions of a weather station can be tested using test cases derived from the sequence diagram.

Visual intuition: a sequence diagram for a weather station shows time running down the page and the participants — SatComms, WeatherStation, Commslink, WeatherData — as vertical lifelines. Arrows trace one request thread: SatComms sends request(report), WeatherStation answers with an acknowledgment and calls reportWeather(), which asks Commslink for get(summary), which invokes summarize() on WeatherData, and the reply send(report) returns to the caller. Each arrow in the diagram names an input and an output, so it tells you exactly which test cases to write: a request input must produce an acknowledgment and eventually a correctly organized report. That is the use case test suite in diagram form.

7.9.4 When to Stop: Policies

For most systems, it is difficult to know how much system testing is essential and when you should stop testing. Exhaustive testing — testing every possible program execution — is next to impossible, so testing has to be based on a subset of possible test cases. Ideally, software companies should have policies for choosing this subset.

These policies might be based on general testing policies, such as a policy that all program statements should be executed at least once. Or they might be based on the experience of system usage, focusing on testing the features of the operational system. Example policies:

• All system functions that are accessed through menus should be tested.
• All combinations of functions accessed through the same menu — like text formatting — must be tested.
• Wherever user input is provided, all functions must be tested with both correct and incorrect input.

7.9.5 Why Automated System Testing Is Difficult

Automated system testing is usually much more difficult than automated unit testing or automated component testing, because system testing involves testing the emergent behavior of the system as well. Automated unit testing relies on predicting the outputs and then encoding these predictions in a program, which is then compared with the result. But the point of implementing a system may be to generate outputs that are too large or cannot be easily predicted — emergent behavior. You may be able to examine an output and check its credibility without necessarily being able to create it in advance, as in test case generation. So automated system test case generation is impossible; automated testing is practical for unit testing and component testing, but not for system testing.

Real-world connection: The weather-station sequence-diagram technique is how modern teams design integration test suites from interaction diagrams, and the "combinations fail, isolated features work" experience explains why large products keep dedicated integration test matrices — combinations of rarely used features (footnotes + multi-column, in the classic example) are exactly what escapes unit-level suites and surfaces only in system-level testing.

7.10 Test-Driven Development

7.10.1 The TDD Cycle

Test-driven development is an approach within development testing, especially at the unit level, in which you interleave testing and code development: you write the test cases before writing the code, run the code without the module being written, watch it fail, write the module, and then it should pass.

Test-driven development was originally introduced as part of the agile development methods. It has now gained mainstream acceptance and may be used in both agile methods and plan-driven development processes, because it is a very effective technique for discovering errors in the requirement specification and correcting them before the code is even written.

The approach: you develop the code incrementally, along with a set of tests for that increment, and you do not start working on the next increment until the code you have developed passes all of its tests.

The fundamental steps, each a process step:

1. Identify the particular increment of functionality that should be developed. This should normally be a small functionality, implementable in a few lines of code.
2. Write a test for this functionality, implemented as an automated test — one that can be executed and report whether it has passed or failed — without writing the functionality.
3. Run the test along with all the other tests already implemented for the system. Initially, since the functionality has not been implemented, the new test fails. This is deliberate: it shows that the new test is adding some value to the test set.
4. Implement the functionality and rerun the test. This may involve refactoring existing code to improve it, and adding new code to what is already there.
5. Once all the tests, including the new tests, run successfully, move on to identifying the new functionality, or the new increment, to implement.

In an automated test environment like JUnit — and the JUnit environment supports regression testing, which is essential for running test-driven development — the code is developed in very small increments, so you have to be able to run every test each time you add a functionality or refactor the program. The tests are embedded in a separate program that runs the tests and invokes the system being tested. Using this approach, you can run hundreds of separate tests in just a couple of seconds.

Test-driven development also helps programmers clarify their ideas of what the code segment is actually supposed to do. If the requirement is ambiguous — not clear — then you cannot write a test case that shows whether the requirement has been met or not met. To write a test, you need to understand what is stated or intended, and this understanding makes it easier to write the required code. Of course, if you have incomplete knowledge or poor understanding of the requirements, test-driven development will not help: if you do not understand the requirements, how do you write a precise test case for that requirement, and you will not be able to develop the required code either. For example, if your computation involves division, you should check that you are not dividing the numbers by zero — if you forget to write a test for this, the checking code will never be included in the program.

7.10.2 Benefits of Test-Driven Development

Beyond better problem understanding, test-driven development brings several benefits.

Code coverage. In principle, every code segment you write should have at least one associated test, so you can be confident that all of the code in the system has actually been executed. The code is being tested while it is being written, so defects are discovered early in the development process. Ensuring that every segment of the code is tested thoroughly is important to discover as many defects as possible while the module or the system is being tested.

Regression testing. Because you develop the test suite incrementally — you write one test case for each requirement — you develop a comprehensive test suite that tests each and every requirement. You can always run the regression test to check that any changes to the program have not introduced new bugs: every time a change, a refactoring, or a new requirement is done, you run the regression test.

Simplified debugging. When a test fails, where the problem lies is usually easy to find — or rather, it should be easy: so far the system has been running well, and when a newly introduced test fails, the problem lies with the newly written code, which should be modified. You need not use exhaustive debugging tools to locate the problem, and in a test-driven environment you rarely need an automated debugger to go through the entire code checking for defects.

Documentation. Test cases written for each requirement act as a form of clear documentation that describes what the code should be doing. Reading the test can make it easier to understand the code: you know the expected input, the behavior of the code, and the expected output of that particular code segment.

Lower regression testing cost. Regression testing involves running the set of tests that were successfully executed after each change is made to the system, to check that the changes have not introduced new bugs and that the new code interacts as expected with the existing code base. Regression testing is expensive and impractical if it is done manually, so you must use automated testing for regression testing: existing tests can be run quickly and cheaply after making a change. In test-first development, all the existing tests must be run successfully before any further functionality is added, so as a programmer you can be confident that the new functionality has not caused or revealed problems with the existing code.

7.10.3 Where Test-Driven Development Is Less Effective

Test-driven development is of significant value in new software development, where the functionality is implemented either in new code or by using components from standard libraries. But:

• If you are using large code components or legacy systems, you need to write tests for these systems as a whole; you cannot easily decompose them into separate testable elements, so incremental test-driven development is impractical.
• Test-driven development may be ineffective with multi-threaded systems, because the different threads may be interleaved at different times in different test runs, and they may produce different test results.
• Even with test-driven development, you still need a system testing process to validate the system — to check that the system meets the requirements of the system stakeholders. System testing also tests performance, stability, and reliability, and checks that the system does not do things it should not do, like producing unwanted outputs.

Test-driven development is now widely used as a mainstream approach to software testing. Most programmers who adopt the approach are happy with it and find it a more productive way to develop software. It is also claimed that test-driven development encourages better structuring of the program and improved code quality, though experiments to verify this claim have been inconclusive.

7.10.4 Student Questions and Answers

Real-world connection: TDD is standard practice across modern professional teams — the JUnit (Java), PyTest (Python), and Jest (JavaScript) ecosystems are built around it, and continuous integration pipelines run the accumulated test suite on every commit, exactly the "hundreds of tests in seconds" workflow the cycle assumes. Its most visible product is the regression suite, which lets teams refactor aggressively because the safety net reruns after every change.

Exam Guidance Summary

• Your exam is on 23 September, and the topics covered until 19 September are included in the syllabus for the semester test — which includes testing primarily, plus all the topics covered earlier. The syllabus will be uploaded with the corresponding chapter references to your textbook as well as the courseware modules of software engineering.
• A first quiz based on the topics covered so far will be uploaded on Canvas soon, with enough time to attempt it before the exams.
• Core distinctions to know well: validation testing versus defect testing — a successful validation test shows the system operates as intended, while a successful defect test makes the system perform incorrectly; verification versus validation — are we building the product right versus are we building the right product, in Boehm's phrasing; and inspections versus testing — static techniques versus execution-based testing.
• Dijkstra's statement — testing can only show the presence of errors, not their absence — is the standard answer for why testing cannot prove a system defect free.
• The equivalence partitioning example, with 4 to 10 inputs and five-digit integers between 10,000 and 99,999, shows the mechanics of boundary values, one midpoint, and adjacent invalid values — a pattern that is easy to revisit numerically.
• Test-driven development is treated as a very important approach: the cycle (test first, watch it fail, implement, watch it pass) and the benefits (code coverage, regression testing, simplified debugging, documentation).
• Release testing, acceptance testing, and the techniques of test case design are covered in detail in the next class.

Key Industry Applications

• Real-world: JUnit for Java, PyTest for Python, and Jest for JavaScript are the automated unit testing frameworks used in everyday professional practice; regression testing is run with them after every change.
• Real-world: mock objects replace slow or unimplemented dependencies in unit tests — standard practice when an object under test calls a database.
• Real-world: pair programming in extreme programming pairs a programmer with a tester as a development-time quality practice.
• Real-world: safety-critical systems, like control software, demand much higher confidence levels than prototypes, so they drive stricter verification and validation budgets.
• Real-world: marketing pressure can force early release — a company may ship before full testing to be first to market, and very cheap software may rely on users as beta testers with lower reliability expectations.
• Real-world: product testing of major software like word processors and spreadsheets follows menu-based policies, because features that work in isolation can fail in combination — the footnote-in-a-multi-column-layout example.
• Real-world: weather station systems use use-case-based system testing, with test cases derived from sequence diagrams.
• Real-world: stress testing message-passing systems with far more messages than expected in practice reveals timing problems, and interface testing with extreme parameter values and null pointers is standard practice for component integration.