A test that cannot fail is not evidence

Test-driven development is usually sold as a design practice: write the test first and you end up with better seams. That is true and it is not the part I care about. The part I care about is narrower and harder to fake. A test earns its place only by being able to fail.

That sounds obvious. It is not, because a suite can be green for three quite different reasons, and only one of them is the good one.

One: the test that never goes red

Hangpolc, the audiobook shelf, has a sleep timer that fades the audio out. It also, until last week, had no volume control. Adding one put two writers on one property: the listener’s level and the fade, both assigned to audio.volume.

The end of the fade did this:

restoreVolume(): void {
  this.el.volume = 1
}

Cancel the timer and you get full volume, at night, next to a sleeping child. The fix was to keep the two apart and give the element their product. The test:

engine.setVolume(0.25)
engine.fadeOut(10)
tick(5)
engine.restoreVolume()
expect(el.volume).toBe(0.25)

Green. So what? A test is a claim that a certain bug cannot be present, and a claim nobody has tried to break is not worth much. So I put the old line back and ran the suite again: two tests failed, and they were the two I had just written. Then I restored the fix and they passed again.

That round trip takes forty seconds and it is the difference between a test and a decoration. I have written tests that passed because a guard clause meant the assertion never ran. One of them probed a coordinate that the structure under test could never produce, so it was not merely untested, it was unsatisfiable. It had been green for days.

If you take one habit from this: break the code on purpose and watch the test go red. Every time. The suite that has never been proven capable of failing is a suite you are trusting on faith.

Two: the test that measures the wrong thing

Rustamp, the Winamp player, was mouse-only for months without anybody noticing, because it did register taps. Playwright’s page.click() synthesises a perfect, stationary mouse sequence, and the browser does the same after a stationary touch. Every test passed.

A finger is not stationary. Driving real touch events through the debugger protocol instead, the failure appeared immediately and it was a cliff, not a slope: up to about fifteen pixels of travel between press and release, Chromium manufactures the compatibility mouse events and everything works. Past that it manufactures nothing at all, and since the app listened for nothing else, the press evaporated before any hit test ran. Under fifteen pixels, flawless. Over it, dead. No test in the suite could see either side of that line, because they all asked the wrong question.

The same trap, one layer down: for a long time I checked the audio by looking at the player’s own spectrum analyser. But the analyser is drawn by the code under test. If the graph is wired wrongly, the analyser is wrong in exactly the same way and agrees with itself beautifully. The honest version patches AudioNode.prototype.connect before boot, taps whatever actually reaches AudioContext.destination, and reads RMS and dominant frequency off that. Every real audio defect in that project was found by the tap and none by the picture.

A test that consults the system’s own opinion of itself is a mirror. Measure at the boundary the user is on the other side of.

Three: the baseline that contains the bug

This one is subtler and it beat four rounds of careful review.

Rustamp gained a gesture that opened a window under the user’s finger, and at certain screen heights that window put a camera button exactly where the finger already was. To check whether a fix worked, reviewers built a control: the same tree with the fix removed, to prove the test could see the defect.

It could not. The control still contained the gesture, the thing that was new, so both builds behaved the same and the hazard was filed as pre-existing twice. It was not pre-existing. Against the actually-deployed version, where the gesture did not exist at all, the same script produced the defect zero times out of twelve, and the working tree produced it every time.

The lesson is not “be careful”. It is structural: your baseline must be the thing you would ship instead, which is production, not a variant of your own change. A control derived by editing the patch shares the patch’s assumptions, and shared assumptions are exactly what a control exists to expose.

What this adds up to

Write the test first if it helps you design. That argument is well made elsewhere. But the reason to keep the discipline is duller and more important. Tests are the only claims in a codebase that are checked automatically, and an unfalsifiable claim checked automatically is worse than no claim at all, because it reads as safety.

Three questions, then, for anything green:

• Have I seen it red, from breaking the code it guards?
• Is it measuring the user’s side of the boundary, or the system’s report of itself?
• If it is a comparison, is the other side something I did not write?

None of the three is expensive. All three have caught something I would otherwise have shipped.