Testing is not a phase. It is a measurement discipline embedded in the engineering process — as invariant to it as dimensional tolerance analysis is to mechanical design.

Measurement as the fundamental act

Physics taught us that the value of a quantity is only meaningful when it is paired with a unit and a measurement procedure. Lord Kelvin’s dictum is unambiguous: „When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.“ The same principle applies without modification to software quality. Quality that cannot be measured is an opinion. Quality that is measured against defined criteria is an engineering result.

Test analysis is the act of deriving measurable test targets from a test basis — requirements specifications, architectural contracts, interface definitions. The test target is the engineering counterpart of the physical measurand: a precisely specified property of the system, for which a pass/fail criterion can be stated. Without this derivation step, testing becomes random exploration, and the results are not reproducible.

Test design: the coverage problem

Coverage theory in testing mirrors sampling theory in experimental science. The space of possible inputs to any non-trivial software system is astronomically large. Exhaustive testing is mathematically infeasible for all but toy systems. The engineer’s task is to select a finite, representative sample that provides statistically sound coverage of the input space with respect to the failure modes of interest.

Equivalence partitioning, boundary value analysis, and decision coverage are deterministic methods for constructing this sample. They are not heuristics — they are systematic procedures derived from a structural or functional model of the system under test. Their application requires that the engineer first hold a sufficiently precise model of the system’s behaviour to partition the input space into equivalence classes. This is test analysis: the cognitive work that precedes test design.

In an agile context, the test analysis is performed iteratively — once per sprint, on the scope of the increment under development. The test basis is the user story or the acceptance criterion, and the coverage criterion is agreed at story definition. In a waterfall-adjacent methodology, test analysis is performed as a structured activity during the requirements specification phase, before the development hot phase begins. The difference is timing, not principle. In both cases, the absence of explicit test analysis produces the same failure mode: tests that explore the tester’s mental model of the system rather than the system’s specified behaviour.

Defect as deviation from specification

A defect is the detection of a deviation between the system’s actual behaviour and its specified behaviour. This definition has two implications that practitioners frequently neglect. First: without a specification, there are no defects — only observations. „It looks wrong“ is not a defect report. „Actual output X deviates from specified output Y under input conditions Z“ is a defect report. Second: the specification is part of the measurement apparatus. A poor specification produces a poor measurement. The quality of the test output is bounded above by the quality of the test basis.

W. Edwards Deming’s analysis of quality in manufacturing systems translates directly: variance in output is a function of variance in process. A test process operating on vague requirements will produce vague defect reports. Defect density numbers extracted from such a process are not engineering measurements — they are artefacts of process noise.

The testing invariant

Testing — as a measurement activity — is invariant across methodologies because the laws of physics are not methodology-dependent. A function either behaves as specified or it does not. The sprint boundary does not change this. The architecture pattern does not change this. The choice between microservices and monolith does not change this. What changes between methodologies is the cadence at which the measurement is performed, and the scope of the test basis available at each measurement point.

The engineering implication: the cost of defect correction rises steeply with the distance between the point of injection and the point of detection. This is not a management principle — it is a consequence of the structure of software artefacts. A defect injected in a requirements document propagates into design, into code, into integration, and into deployment before detection. At each propagation step, it acquires co-dependent artefacts that must also be corrected. The economic case for early test analysis — shifting measurement left — follows from this structure.

References