The Repo Was Real. The Tests Didn't Know That.

The thing your test suite has never had to think about

Somewhere in what you run, there is a test suite that has only ever executed in one place: your CI runner. It checks out a fresh copy of the repo, does whatever it wants to that copy, and throws it away when the job ends. You didn’t design the suite around that fact. You didn’t have to. CI just always hands you a room nobody’s lived in yet, so nothing in the tests ever had a reason to ask whose room it was.

That’s fine, right up until the day the same suite runs somewhere that isn’t CI. A laptop. A box with a real checkout on it. Your actual working copy, with your actual history in it. The suite doesn’t know the difference, because you never told it there was one to check for. It was never asked to.

You have this somewhere. Almost every test suite that “just runs in CI” is quietly leaning on a property it never declared and nothing enforces.

What ours actually did

We run a set of shell and Ruby suites that exercise git-adjacent tooling: branch handling, worktree setup, commit authorship, that kind of thing. Written and proven correct against exactly one environment, the disposable checkout CI always gave them.

Someone ran the full suite locally, inside the live vault, to check something unrelated. The fixture code did what it always does: it pointed refs/heads/main at a fixture commit, spun up four branches named feat/real* (an honest name, in hindsight, for the wrong reason), and ran git config user.name t against whatever repo it found itself in. It found the real one.

None of that alone would have mattered if it had stayed on a laptop. It didn’t. A scheduled sync job ran a few minutes later, saw local commits it didn’t recognise as fixtures, and pushed. Forty-one of them. To the real GitHub repo. main came out the other side with a tree that had deleted most of the vault, and the identity fixture had already stuck - the next real commit after the incident was authored t <t@t>, which is now permanently part of our history, because rewriting published commits is a bigger call than leaving one odd author line as a scar.

For about two minutes, our own commit history had no vault in it at all. In our defence, nobody was watching that sync job closely enough to catch it in those two minutes. In our further defence, there was no reason to be. It had run the same way, correctly, every day before this one.

Why nothing complained on the way there

Here’s the part worth sitting with: the tests were never wrong. Every assertion in that suite was checking the right thing and getting the right answer, for the environment it was built for. git config user.name t inside a throwaway checkout is a completely reasonable thing for a fixture to do. It’s supposed to be discarded thirty seconds later.

The bug wasn’t in any single line. It was in the gap between “this suite is correct” and “this suite is safe to run” - two claims that look identical until the second one turns out to depend on something the first one never mentioned. A test can pass every assertion you wrote for it and still be one working directory away from doing real damage, because passing only tells you the code did what it does. It tells you nothing about what it assumes about where it’s standing.

If you’ve ever taken a script built for one context - a migration written for a scratch database, a cleanup job written for a scoped bucket, a suite written for CI - and run it “just quickly” somewhere else because it seemed harmless, you’ve made this same bet. Most of the time you win it, because most fixtures don’t happen to overlap with anything real. That’s not the same as the bet being safe.

So what actually makes a test hermetic?

Not “it doesn’t call the network.” Not “it uses fixtures instead of real data.” A test is hermetic when it builds the room it needs instead of assuming one will be there - when the disposability isn’t a property of wherever you happen to run it, but a property the test creates for itself, every time, regardless of what’s lying around.

That’s what we rebuilt. The suite runner now clones into a fresh directory with no origin remote at all, so there is nothing for a fixture commit to reach even if it wanted to, and redirects HOME so identity fixtures land somewhere that isn’t your real git config. Then it checks, after every suite, whether anything managed to escape anyway.

That last check turned out to be its own small lesson. The obvious version compares ref values before and after - did main move - and it’s wrong, because in a live system main moves on its own constantly; our vault’s own scheduled jobs shift it several times an hour. A check built on values flagged suites that had done nothing wrong. The one that actually works watches for ref names and identity strings that shouldn’t exist in a disposable clone at all - not “did something change”, but “did something appear that could only have come from a fixture.” Measuring the wrong thing gives you false alarms; measuring the right thing gives you a check that stays quiet on a normal day and loud on the day that matters.

The check to run against your own setup

Find the thing you run that was written assuming a property of its usual home - a scratch directory, a disposable database, a sandboxed account, a checkout nobody will keep. Then ask, honestly, whether that property is something the code creates, or something it just always happened to receive.

If it’s the second one, you don’t have a hermetic test. You have a test that has been lucky about where it’s been run so far. The difference between those two only ever shows up on the day someone runs it somewhere it wasn’t built for - and by then it isn’t a question anymore, it’s a restore.