The forecast was wrong, and everyone knew it was wrong, and nobody knew why.
That’s the specific, uncomfortable place to start a debugging session from. Not “there’s a bug” — there’s a number, printed on a dashboard people actually look at, and it’s been quietly too high for weeks. Not broken enough to alarm anyone. Wrong enough that nobody trusted it.
I went looking for the model. That’s the first mistake, and I want to be honest about it: I assumed there was one forecasting system, and it had a bug in it. What I actually found, once I started reading the code instead of the documentation about the code, was two separate systems living in the same codebase. One was a machine learning model — trained, evaluated, and then never actually saved anywhere. A path that looked live in the architecture diagram and had been dead for who knows how long. The other was a much simpler statistical system, tracking completion ratios against historical patterns, retrained regularly, and — when I actually tested it in isolation — producing numbers that looked correct.
That was the surprise. The model everyone assumed was doing the forecasting wasn’t running at all. The real system was quietly fine. Which meant the inflated number wasn’t coming from a broken forecast. It was coming from something downstream of a correct one.
The struggle wasn’t finding the second system — it was sitting with the discomfort of “the thing I was sure was broken turned out to be working,” and having to admit that meant the actual bug was somewhere I hadn’t looked yet. It’s a specific kind of frustrating to disprove your leading theory a week into hunting for something. The instinct is to keep pushing on the theory because you’ve already invested in it. I had to let it go and start over from “okay, if the inputs are right, where does the number actually go wrong.”
The lead I ended up with, and haven’t fully closed yet: the projection math likely divides a current count by a ratio measured at a specific hour, and if that ratio is underestimated early in a shift, the division inflates everything downstream of it — a small early error compounding into a big late one. I don’t have it fully proven yet. But it’s a real, specific, testable hypothesis, which is further than “the forecast is wrong” ever got anyone.
The transferable part isn’t the bug. It’s that “which system is actually running” is a question worth asking before “what’s wrong with the system,” every time — because the two questions send you down completely different paths, and only one of them is real.
Next: instrumenting the actual division step directly, hour by hour, instead of trusting the summary numbers on either end of it.
Leave a Reply