Twenty-Five Successful Runs. Zero Posts Shipped.

You’ve probably got one of these already

Somewhere in what you run there’s a job that rebuilds its plan from nothing, every single time it fires. A scheduler that looks at what’s available right now and ranks the best options. A sync script that re-reads a source and decides what to push next. A recommendation queue that scores candidates fresh on every tick.

It feels stateless, and stateless sounds like a virtue. No leftover state to drift, no old records to clean up, just a clean decision made from the current facts each time it runs.

Here’s the bit that isn’t a virtue. If the job has no memory of what it tried last time, it also has no memory of what got turned down last time. So it offers the same rejected thing again. And again. As far as the job is concerned, every run is the first run, forever, until something outside it says otherwise.

Ask yourself what happens in your version of this when the thing it offers gets rejected downstream. Does anything remember that happened tomorrow morning?

What ours did, every morning, for over a week

We run a job that picks the best content we have each morning and queues it to LinkedIn through Buffer. It looks at what’s available, scores it, and offers the top few.

Buffer checks every post against what it already holds and refuses duplicates. That’s Buffer doing its job correctly. Our job logged each refusal. That’s our job doing its job correctly too.

Nobody had told the ranking step that “refused yesterday” should count against a post today. So the same small handful of top-scoring posts kept coming back up, got refused again, and got logged as refused again. Twenty-five mornings of that, back to back, and the page went quiet the entire time.

In our defence, we did try. Twenty-five times.

Every individual piece behaved exactly as designed. That is what made the whole thing invisible - there was no error anywhere to go looking for.

Why nobody caught it for over a week

The daily message read “queued to Buffer” with a count next to it, and the count was real. It counted the local file we’d just built and were about to attempt sending. It said nothing about how many of those attempts actually landed, because that number lived one layer downstream and nothing carried it back up.

So every morning produced a cheerful, accurate, completely misleading sentence. Accurate about the file we made. Misleading about what happened to it. A number that is true and useless is worse than no number at all, because it’s precisely the thing that stops anyone from checking further.

Rejected and failed read identically from the outside, if the only thing being watched is whether the job finished without throwing an error. Ours finished clean twenty-five times running.

The fix wasn’t a smarter queue, it was a shorter memory gap

The tempting fix is to make the ranking cleverer, so it stops picking weak candidates. That wasn’t the actual fault. The ranking was fine on its own terms. What was missing was the loop back: nothing told the ranking step that the thing it had just offered had already been tried and turned down.

So it keeps a ledger now. Every attempt and its outcome, checked before the next run picks its favourites. Nothing about the scoring changed. The queue simply stopped being allowed to forget what it had already been told no to.

The daily message changed too. It now reports what the downstream system actually confirmed, not what the local file says was attempted. If that number comes back at zero for the wrong reason, that is now a fact the message is built to show rather than paper over with a cheerful count of something else.

If a job in your own stack reports success on every run, it’s worth one afternoon checking whether “success” and “did the thing” are actually the same number. Ours weren’t, for the best part of two weeks, and every single run said they were.