The forecast had been “very inflated” for weeks. Nobody could say by how much, or why — just that the projected end-of-day numbers didn’t match what actually happened, often enough that people had started mentally discounting them.
I went looking for the model first, because that’s where you look. What I found instead was that the fifteen-minute data feeding the forecast was cumulative — a running total for the day, not a fresh count per interval. I confirmed it the boring way: pulled the raw rows and watched them climb, strictly, all day — never dropping, only ever adding on top of the last number. That’s the signature of a running total, not a series of separate readings.
The actual bug wasn’t in that data. It was one step downstream, in the code that consumed it. Somewhere in the pipeline, someone had written a loop that summed those cumulative numbers as if they were fresh increments — adding a running total to another running total to another, compounding a small mistake into a large one. In one function I found the two clearest evidence of it happening live: one number in the loop correctly took the maximum value across the period, and the number right next to it — same loop, same author, same line count away — used addition instead. One field right. One wrong. Nobody had noticed, because both numbers looked plausible in isolation.
Before I told anyone what I’d found, I wrote down what the bug should produce if I was right — a specific projected number, checkable against the live dashboard within the hour. I did the math, then went and looked. The dashboard read almost exactly what I’d predicted it would if the bug was real: roughly double the actual count, growing toward quadruple by late afternoon as more cumulative snapshots piled onto the sum.
The struggle wasn’t finding the bug. It was resisting the urge to declare victory the moment I found *a* plausible cause, instead of confirming it actually explained the whole shape of the problem — the way it got worse through the day, not just that it was wrong. A bug that only explains part of a symptom isn’t the bug yet.
The lesson: if you can predict a specific number a bug should produce, and go check it against reality before you tell anyone you found the answer, you’ve turned a guess into a proof. That one habit is the difference between “I think I found it” and “I found it.”
Next: fixing the aggregation at its actual source, not patching the number that comes out the other end.