For about a week, a thing I built looked dead.
Some background. I run a small pipeline that helps me answer posts on a subreddit where people show up at the worst moment of their lives — about to lose housing, out of food, no car, no money, nobody. A collector watches the feed. A triage step reads each post and decides: is this a real, findable need with enough detail to actually help, or not. The ones that qualify land in a queue. I get a buzz. I do the resource lookup by hand — the real shelter, the real intake hours, the real number that still works — and I write back myself, in my own words. The machine never posts. It just finds the people I’d otherwise miss and does the tedious part of the homework.
For a week, no buzz. Empty queue on my phone, empty queue on the laptop. The obvious read is: it broke. Something upstream died and the whole thing went quiet. I had a super busy week and kept checking but nothing.
So I started pulling it apart cause today I had some free time.
First I figured the feed reader had stopped reading. It hadn’t — it was pulling posts every fifteen minutes, clean, no errors. Then I figured the database it writes to had been wiped, because the one I looked at was empty. Wrong database — the empty one was a decoy sitting in a folder, and the real one was somewhere else entirely, fat and healthy and full of data. Then I figured the triage step had gone haywire and was rejecting everyone. Nope — I pulled its decision log and it was scoring posts correctly, that day, that hour. Real people, real verdicts, all right. Then I figured a power blip that had bounced the machine that morning was the culprit. Also wrong — the problem was nine days old and the reboot was that morning. I was pattern-matching to the most dramatic explanation available and dressing up each guess as a conclusion.
Nothing was broken. The pipeline had been working the entire week. It had quietly triaged over a hundred people into the queue. The *page that shows the queue* only displays the top three, oldest first, with no count anywhere on it. And a change I’d made the week before — loosening the filter so it would help food-need posts and not just eviction posts — meant a lot more people were qualifying. The queue went from a couple dozen to over a hundred. So the same three oldest posts sat frozen at the top of the page for nine days while ninety-odd newer ones stacked up behind them, invisible, because the page had no way to say “there are more of these.” A full room and an empty room look identical through a keyhole that only shows the doorway.
That’s the whole bug. Not a crash. A window that couldn’t show its own depth, and me not noticing because the window looked the same either way.
The fix was two moves. Close out the pile of stale ones that had aged past usefulness — close, not delete, so it’s all reversible if I got the call wrong. And then a permanent rule: posts age out of the visible queue after two days on their own, so it can never silently fill up and hide the live ones again. The deeper lesson — that a page which shows a fixed few with no count can hide a problem for a week — is the kind of thing you only learn by getting bitten.
Two notes on the debugging itself, for anyone who does this work.
One: I kept guessing ahead of the evidence. Every wrong theory came from writing a conclusion before I’d read the thing that would confirm or kill it. The move that finally worked was dead simple and I should’ve led with it — stop theorizing, ask the running program which file it actually has open, read the log it actually wrote, check the seam where the data actually stops. The system will tell you what’s wrong if you read it instead of narrating it.
Two: I was working with an AI assistant on this, and it was having a rough day — the service behind it was degraded, and it showed. It kept handing me confident wrong theories, same as I was generating on my own. What made that workable instead of a disaster is a standing rule between us: when one of us is off, the other says so, out loud, and we slow down. I told it flat out it had been wrong four times and to stop sounding sure. It did. That’s not an AI story. That’s the same rule that keeps two people from driving a bad decision off a cliff because neither wanted to be the one to say “wait.” The tool being fancy doesn’t change the discipline. If anything it raises the stakes on it.
Anyway. The queue’s healthy again, the people in it are getting answered, and the thing that looked dead for a week was alive the whole time, just talking through a window too small to hear it. Fixed the window.
Back to it.