Your unauthenticated test client is probably authenticated

---

Our suite was green for months while three separate things in it were not testing what they said they were.

None of them was a flaky test or a bad assertion. Every one was structural, and no individual test was in a position to notice, which is why they lasted.

One of them shipped real defects underneath it for months. Each now has a small, ugly test against it, so the suite has to keep telling the truth about itself instead of being trusted to.

Our shared test fixture `api_client` was documented as unauthenticated. Thirty-two assertions across nineteen files used it to prove that an anonymous caller gets a 401 or a 403.

It was not unauthenticated.

The client's `login()` method calls `set_auth_token()` on itself. It mutates the instance it runs on. And the session-scoped `auth_token` fixture called `api_client.login(...)` to get its token, so from the moment any test in the run pulled the authenticated client, the shared "unauthenticated" client was carrying a bearer token for the rest of the session.

Those thirty-two assertions still passed.

They checked that a request was refused. The request *was* refused, for an unrelated reason or because the endpoint would have refused an authenticated caller too, and nobody was looking at which.

`test_00_fixture_contracts.py`, which asserts nothing about the platform and everything about the fixtures.

Is the unauthenticated client actually unauthenticated, at this point in the session, after the other fixtures have run?

The rule falls out of it. If a fixture's *name* is load-bearing for a security assertion, that fixture needs a test of its own.

Ours did not have one.

The second one we have written up before, in part 1 of this series, so here it is in three lines.

A comment in one suite delegated saga-compensation coverage to another test file by name. That file had not existed since May 2026 and its source was never committed. Fourteen broken compensation templates shipped underneath the comment while every run reported green.

The guard we built for it, `test_00_coverage_claims.py`, scans test modules for prose that delegates coverage elsewhere and fails if the cited module exists nowhere in the tree. It asserts nothing about the platform. It asserts that the suite's claims about itself are checkable, on the principle that a dangling delegation is worse than no comment at all, because it stops the next reader from looking.

Both defects have the same shape as the fixture one. Something the suite said about itself was false, and no assertion inside any individual test was in a position to notice.

The replacement for the vanished saga test reads the real manifest files off disk and checks that every declared inverse operation names fields that actually exist in the operation it is inverting.

Some of its tests fail today, and each failure names a declaration that does not line up with the operation it describes.

We did not fix the fourteen and then ship the test.

Shipping it red is the point. A suite that only ever goes green is a suite that can be made green by deleting an assertion, and this one is a live counter of a known debt that anybody on the team can read without asking who owns it.

Its header describes three anti-vacuity measures that are worth copying. It asserts a floor on the number of manifests it found, so a glob that silently matches nothing fails instead of passing trivially. It contains zero skips. And it carries a test that fails if the template syntax ever changes, because the regex it uses to find template references would otherwise quietly match nothing and report a clean bill of health forever.

That last one is the subtle one.

A validator whose pattern stops matching does not throw. It reports zero problems, which looks exactly like success.

The same idea sits under each fix: **a test should be able to fail for the reason it is named after.**

We now write some tests in three parts when a feature is mid-flight. A characterisation test that passes today and documents the current, imperfect behaviour. A guard that must stay green. And a discriminator that is red until the feature lands, written so that it skips rather than false-fails while the platform lacks the field it needs, which means it activates by itself on the day the build ships.

Several files carry a banner saying they have not yet been executed against a live platform, with an instruction to treat a first failure as "the fixture is wrong" before "the platform is wrong". That instruction has been correct more often than not.

None of this catches a test that exercises the right code and asserts something too weak. A meta-test can check that a cited file exists. Whether the assertion inside it is worth making remains a judgement someone has to make by reading it.

It also costs something real. Three test files in our suite now test the suite, not the product, and a reviewer is entitled to ask what they are buying. The answer is the fourteen shipped defects and the thirty-two vacuous assertions, both of which were found by writing exactly these files.

And some of our composite coverage is still a skipped placeholder rather than a test.

That is at least visible in the file now, rather than hidden behind a comment pointing somewhere else. It is an improvement on where we started. It is not the same as being covered, and we would rather you heard that from us.

Grep your test suite for the names of other test files, wherever prose can hide them. Check each one resolves to something on disk.

That is a shell one-liner. It is how we found the first problem.

Then take the fixture your security assertions depend on. Write one test that checks it is what its name says it is, at the point in the run where it is actually used.

If it passes, you have lost twenty minutes. If it fails, you have learned that some number of your security tests have been decorative, and you now get to find out which.

*(Disclosure: I work on Supero. Every defect described above is ours, and was found by us.)*