Robert Dugger — Dev Blog

Tag: debugging

  • The Rule Said 0.25%. The Math Said It Was Actually Enforcing 0.056%.

    There was a rule that had been running for a long time: if a list’s contact rate drops below a fixed number after enough attempts, cut it and move to something else. Simple, defensible-sounding, the kind of rule nobody questions because it’s been there since before anyone currently on the team arrived.

    I sat down to actually check what that rule was defensible *against* — not what it claimed, what it proved. A flat cutoff at a fixed percentage doesn’t account for how much you can trust an observation at a given sample size. Cut a list at 400 attempts and 0.25%, and the honest, statistically rigorous floor that observation actually clears — accounting for the real uncertainty at that sample size — turns out to be 0.056%. Not 0.25%. Seven times lower. The rule’s name promised one standard and delivered a much more trigger-happy one, and nobody could see the gap because nobody had run the number.

    That explained something that had been bothering people for longer than the rule itself had existed: cuts kept getting reversed. A list would get pulled for underperforming, and later turn out to have been fine. I went and checked, against the real history, how often that happened — pulled every moment the rule would have fired across a quarter of real data, then checked what those same lists did in the following day. About one in six to one in four of them recovered on their own within 24 hours. Not because the rule was wrong to exist. Because it had never been calibrated to know the difference between “actually bad” and “noisy this hour.”

    While I was in that part of the system, I went looking for the audit trail — the log of every automated decision the balancer had ever made, expecting to be able to reconstruct exactly which lists got swapped for what reason. The log existed. It had the right columns for it — which list got removed, what replaced it. Every single row had those columns empty. Every reason field said the same generic string, verbatim, on all thirty-four thousand rows. The audit trail had been built and never actually wired up to record anything real.

    The struggle in both of these wasn’t technical — it was resisting the instinct to fix the surface symptom (adjust the cutoff number) instead of asking whether the whole shape of the rule was the problem. A better number on a badly-shaped rule is still a badly-shaped rule.

    The lesson: when a threshold has a specific number in its name, ask what that number is actually defensible against, not what it claims to be. The two are not always the same thing, and the gap between them is where false confidence lives.

    Next: replacing the flat cutoff with a rule that adjusts automatically for how much history a list has behind it, instead of hand-picking a new number every time the old one stops working.

  • The Number Was Wrong by 2x, and I Found It by Predicting the Wrong Number in Advance

    The forecast had been “very inflated” for weeks. Nobody could say by how much, or why — just that the projected end-of-day numbers didn’t match what actually happened, often enough that people had started mentally discounting them.

    I went looking for the model first, because that’s where you look. What I found instead was that the fifteen-minute data feeding the forecast was cumulative — a running total for the day, not a fresh count per interval. I confirmed it the boring way: pulled the raw rows and watched them climb, strictly, all day — never dropping, only ever adding on top of the last number. That’s the signature of a running total, not a series of separate readings.

    The actual bug wasn’t in that data. It was one step downstream, in the code that consumed it. Somewhere in the pipeline, someone had written a loop that summed those cumulative numbers as if they were fresh increments — adding a running total to another running total to another, compounding a small mistake into a large one. In one function I found the two clearest evidence of it happening live: one number in the loop correctly took the maximum value across the period, and the number right next to it — same loop, same author, same line count away — used addition instead. One field right. One wrong. Nobody had noticed, because both numbers looked plausible in isolation.

    Before I told anyone what I’d found, I wrote down what the bug should produce if I was right — a specific projected number, checkable against the live dashboard within the hour. I did the math, then went and looked. The dashboard read almost exactly what I’d predicted it would if the bug was real: roughly double the actual count, growing toward quadruple by late afternoon as more cumulative snapshots piled onto the sum.

    The struggle wasn’t finding the bug. It was resisting the urge to declare victory the moment I found *a* plausible cause, instead of confirming it actually explained the whole shape of the problem — the way it got worse through the day, not just that it was wrong. A bug that only explains part of a symptom isn’t the bug yet.

    The lesson: if you can predict a specific number a bug should produce, and go check it against reality before you tell anyone you found the answer, you’ve turned a guess into a proof. That one habit is the difference between “I think I found it” and “I found it.”

    Next: fixing the aggregation at its actual source, not patching the number that comes out the other end.

  • Two Forecasting Systems, and Only One of Them Was Real

    The forecast was wrong, and everyone knew it was wrong, and nobody knew why.

    That’s the specific, uncomfortable place to start a debugging session from. Not “there’s a bug” — there’s a number, printed on a dashboard people actually look at, and it’s been quietly too high for weeks. Not broken enough to alarm anyone. Wrong enough that nobody trusted it.

    I went looking for the model. That’s the first mistake, and I want to be honest about it: I assumed there was one forecasting system, and it had a bug in it. What I actually found, once I started reading the code instead of the documentation about the code, was two separate systems living in the same codebase. One was a machine learning model — trained, evaluated, and then never actually saved anywhere. A path that looked live in the architecture diagram and had been dead for who knows how long. The other was a much simpler statistical system, tracking completion ratios against historical patterns, retrained regularly, and — when I actually tested it in isolation — producing numbers that looked correct.

    That was the surprise. The model everyone assumed was doing the forecasting wasn’t running at all. The real system was quietly fine. Which meant the inflated number wasn’t coming from a broken forecast. It was coming from something downstream of a correct one.

    The struggle wasn’t finding the second system — it was sitting with the discomfort of “the thing I was sure was broken turned out to be working,” and having to admit that meant the actual bug was somewhere I hadn’t looked yet. It’s a specific kind of frustrating to disprove your leading theory a week into hunting for something. The instinct is to keep pushing on the theory because you’ve already invested in it. I had to let it go and start over from “okay, if the inputs are right, where does the number actually go wrong.”

    The lead I ended up with, and haven’t fully closed yet: the projection math likely divides a current count by a ratio measured at a specific hour, and if that ratio is underestimated early in a shift, the division inflates everything downstream of it — a small early error compounding into a big late one. I don’t have it fully proven yet. But it’s a real, specific, testable hypothesis, which is further than “the forecast is wrong” ever got anyone.

    The transferable part isn’t the bug. It’s that “which system is actually running” is a question worth asking before “what’s wrong with the system,” every time — because the two questions send you down completely different paths, and only one of them is real.

    Next: instrumenting the actual division step directly, hour by hour, instead of trusting the summary numbers on either end of it.

  • The Bug That Looked Like Slow, and Was Actually Broken

    It was one of those checks that should’ve taken thirty seconds. I ran a search against a real list — a few hundred leads, standard call, nothing exotic — and it just sat there. No error. No result. Just quiet.

    I assumed it was slow. I’d built the thing to hit an internal API and page through records, so “slow” was the obvious story, and I believed it for longer than I should have. I even started looking at whether I needed to add caching.

    Then I ran the same search on a smaller list — thirty records instead of three hundred — and it worked instantly. That’s when I knew it wasn’t slow. Slow doesn’t have a cliff. Broken does.

    The real problem was a single field. My data model marked customer email as an optional, validated email field — which sounds correct, and is correct, right up until the source system’s convention for “no email on file” turns out to be an empty string instead of a null. Pydantic’s email validator doesn’t know what to do with an empty string. It doesn’t skip it. It rejects it. And it rejects it silently enough, deep enough in a batch operation, that the whole search just — stopped. No traceback pointing at the actual cause. Just nothing.

    I’d been debugging the wrong problem for the better part of an hour. I was optimizing for a diagnosis I’d made before I had any real evidence for it, and once I’d said “it’s probably slow” out loud, I kept looking for reasons that were true instead of reasons that were right.

    The fix was small once I found it — one validator that runs before the email check, converting empty strings to null so the real validation logic still applies to anything that’s actually malformed. Seven new tests to make sure it stayed fixed. But the fix isn’t the lesson. The lesson is that “it’s slow” and “it’s broken” produce completely different debugging paths, and picking the wrong one costs you real time before you even notice you’re on it.

    I’ve started treating my own first explanation as a hypothesis to disprove, not a starting point to build on. The five-minute version of that discipline: before you optimize anything, prove it’s actually the bottleneck you think it is.

    Next: going back through every other endpoint in the same tool with the same question — not “is this slow,” but “have I actually confirmed that, or just assumed it.”

← rfditservices.com