Nothing Errored. That Was The Bug.

The step you have that never throws

Somewhere in what you run, there is a piece of code that reads a record, changes something about it, and writes it back out. A migration. A webhook handler. A CI step that pulls an issue or a PR and republishes it in a shape something else can use.

You tested it once. It ran. Nothing crashed. You called it done.

“Didn’t crash” is doing more work in that sentence than it deserves. It tells you the shape you checked survived. It tells you nothing about the fields you didn’t check, because you weren’t looking for them, because you had no reason to be.

Here’s the actual rule: whatever a transform step doesn’t explicitly carry forward is not guaranteed to survive it. Not “might get corrupted” - just gone, silently, indistinguishable from a value that was never set in the first place. Nothing downstream complains, because the code reading the output only ever asks “is this field here”, never “should it have been”.

You have this somewhere. I would bet on it.

Ours turned up by accident

We run a job that scans open issues and PRs across our repos every morning, classifies them, and writes a ranked snapshot. Downstream of that, a separate gate decides whether a PR is safe to merge on its own - and it only ever says yes if the PR is carrying one specific label.

A re-scan fired 46 minutes after the morning run, for a completely different reason: checking whether the backlog had moved since the first pass. Nothing on GitHub had changed in those 46 minutes, so the two runs should have produced identical output. They didn’t. The second run’s PR records came back with their labels empty. The first run’s didn’t.

Same PRs. Same GitHub state. Different output. That’s the one shape of bug you can only see with two things sitting next to each other - a single run just looks like a result.

The cause, once we went looking, was almost boring: the classification step rebuilt each PR record from the handful of fields its ranking logic actually cared about, and labels wasn’t one of them. So it never made it into what got written out. The step did exactly the job it was written to do. It just wasn’t the job the merge gate, three files away, was quietly counting on it to also do.

No PR happened to be carrying the safety label that particular morning, so nothing actually merged that shouldn’t have. That is the only reason this is a story and not an incident report.

Why the obvious fix isn’t the real one

The tempting fix is to write a test: assert that labels survives classification. Ship it, move on.

That test would have caught this exact bug. It would have told you nothing about the next one - whatever field gets added next month that nobody remembers to carry through either, because the same rebuild-from-scratch pattern is still sitting there, waiting for someone to add a field to the input and forget to add it to the output.

A test for one field protects one field. What actually needed fixing was the assumption that a step “worked” because it ran clean, when running clean and preserving everything you didn’t explicitly touch are two different claims and only one of them was ever checked.

The fix that generalises is structural, not a value you assert on: compare a step’s output against its input’s field set, or replace whole-record rebuilds with narrow edits that start from a full copy and only override what actually changed, instead of reassembling a record from the fields you remembered to ask for. Either way, the step stops being able to shrink its own output without saying so.

What to do with this

If a step in what you run has only ever been checked against itself - one pass, watched for errors, called good - you don’t know it preserves anything you didn’t explicitly test for. You know it doesn’t crash. Those are not the same fact, and the second one is the one that actually matters when something downstream is quietly depending on a field you’ve never once thought about.

The only way we found ours was luck: a second run existed, for a reason that had nothing to do with this bug, and something happened to look at the difference. Worth doing on purpose, rather than waiting on a scheduler to hand it to you by accident.