Robert Dugger — Dev Blog

Tag: debugging

  • Every Room Looked Different, and That’s Exactly What Told Me They Were All the Same Bug

    I’d asked an agent to verify something specific about a puzzle solver I’d been building: were the “dead zones” — tiles a player could never actually reach or use — genuinely different from room to room, or was something wrong. The report came back clean. Dead zones varied by room, it said, consistent with rooms actually being different from each other. I almost accepted that and moved on.

    Something about it nagged at me. The dead zones it reported were all clustered in roughly the same place relative to the start position, room after room, regardless of how differently each room was actually laid out. Different rooms, suspiciously similar shapes of “unreachable.”

    I went and read the actual search function instead of trusting the summary. It was supposed to explore every tile reachable from the start, treating locked gates as walls — you can’t walk through a locked gate — but letting the player pass through unlocked ones freely, the same way every other piece of movement logic in the codebase already correctly handled it. This one function didn’t. It blocked *every* gated tile, locked or unlocked, treating an open door the same as a solid wall. The search never got past the first gate in any room, trapped in whatever small pocket happened to be reachable before hitting one. That’s why every room’s “dead zones” looked similar — they weren’t dead zones at all, they were just everything past the nearest door, misreported as unreachable because the search itself couldn’t reach it.

    The part that actually mattered wasn’t the bug — it’s that the bug produced a report that looked correct on its own terms. Varied dead zones per room is exactly what you’d expect from working code. The only reason I caught it was a vague sense that the variation looked too similar to be real variation, which isn’t a rigorous test, it’s a hunch, and I almost didn’t follow it.

    While I was in that function fixing it, I found a second, smaller version of the identical mistake nearby: one type of interaction correctly filtered out tiles that couldn’t use it before attempting anything, and a second, similar interaction type had no equivalent filter at all — it just tried every tile and let a later step silently discard the ones that didn’t work. Same root cause, same shape, different feature.

    The struggle was trusting a summary that had every surface property of a correct answer. Nothing about the report was implausible. That’s what made it dangerous.

    The lesson: “the results look varied and plausible” is not the same claim as “the results are correct,” and a hunch that something’s too tidy is worth five minutes of reading the actual code before you accept it.

    Next: the same read-the-actual-function discipline, applied to the two other suspected shortcuts in the same solver before trusting any of its output again.

  • The Rule Said 0.25%. The Math Said It Was Actually Enforcing 0.056%.

    There was a rule that had been running for a long time: if a list’s contact rate drops below a fixed number after enough attempts, cut it and move to something else. Simple, defensible-sounding, the kind of rule nobody questions because it’s been there since before anyone currently on the team arrived.

    I sat down to actually check what that rule was defensible *against* — not what it claimed, what it proved. A flat cutoff at a fixed percentage doesn’t account for how much you can trust an observation at a given sample size. Cut a list at 400 attempts and 0.25%, and the honest, statistically rigorous floor that observation actually clears — accounting for the real uncertainty at that sample size — turns out to be 0.056%. Not 0.25%. Seven times lower. The rule’s name promised one standard and delivered a much more trigger-happy one, and nobody could see the gap because nobody had run the number.

    That explained something that had been bothering people for longer than the rule itself had existed: cuts kept getting reversed. A list would get pulled for underperforming, and later turn out to have been fine. I went and checked, against the real history, how often that happened — pulled every moment the rule would have fired across a quarter of real data, then checked what those same lists did in the following day. About one in six to one in four of them recovered on their own within 24 hours. Not because the rule was wrong to exist. Because it had never been calibrated to know the difference between “actually bad” and “noisy this hour.”

    While I was in that part of the system, I went looking for the audit trail — the log of every automated decision the balancer had ever made, expecting to be able to reconstruct exactly which lists got swapped for what reason. The log existed. It had the right columns for it — which list got removed, what replaced it. Every single row had those columns empty. Every reason field said the same generic string, verbatim, on all thirty-four thousand rows. The audit trail had been built and never actually wired up to record anything real.

    The struggle in both of these wasn’t technical — it was resisting the instinct to fix the surface symptom (adjust the cutoff number) instead of asking whether the whole shape of the rule was the problem. A better number on a badly-shaped rule is still a badly-shaped rule.

    The lesson: when a threshold has a specific number in its name, ask what that number is actually defensible against, not what it claims to be. The two are not always the same thing, and the gap between them is where false confidence lives.

    Next: replacing the flat cutoff with a rule that adjusts automatically for how much history a list has behind it, instead of hand-picking a new number every time the old one stops working.

  • The Number Was Wrong by 2x, and I Found It by Predicting the Wrong Number in Advance

    The forecast had been “very inflated” for weeks. Nobody could say by how much, or why — just that the projected end-of-day numbers didn’t match what actually happened, often enough that people had started mentally discounting them.

    I went looking for the model first, because that’s where you look. What I found instead was that the fifteen-minute data feeding the forecast was cumulative — a running total for the day, not a fresh count per interval. I confirmed it the boring way: pulled the raw rows and watched them climb, strictly, all day — never dropping, only ever adding on top of the last number. That’s the signature of a running total, not a series of separate readings.

    The actual bug wasn’t in that data. It was one step downstream, in the code that consumed it. Somewhere in the pipeline, someone had written a loop that summed those cumulative numbers as if they were fresh increments — adding a running total to another running total to another, compounding a small mistake into a large one. In one function I found the two clearest evidence of it happening live: one number in the loop correctly took the maximum value across the period, and the number right next to it — same loop, same author, same line count away — used addition instead. One field right. One wrong. Nobody had noticed, because both numbers looked plausible in isolation.

    Before I told anyone what I’d found, I wrote down what the bug should produce if I was right — a specific projected number, checkable against the live dashboard within the hour. I did the math, then went and looked. The dashboard read almost exactly what I’d predicted it would if the bug was real: roughly double the actual count, growing toward quadruple by late afternoon as more cumulative snapshots piled onto the sum.

    The struggle wasn’t finding the bug. It was resisting the urge to declare victory the moment I found *a* plausible cause, instead of confirming it actually explained the whole shape of the problem — the way it got worse through the day, not just that it was wrong. A bug that only explains part of a symptom isn’t the bug yet.

    The lesson: if you can predict a specific number a bug should produce, and go check it against reality before you tell anyone you found the answer, you’ve turned a guess into a proof. That one habit is the difference between “I think I found it” and “I found it.”

    Next: fixing the aggregation at its actual source, not patching the number that comes out the other end.

  • Two Forecasting Systems, and Only One of Them Was Real

    The forecast was wrong, and everyone knew it was wrong, and nobody knew why.

    That’s the specific, uncomfortable place to start a debugging session from. Not “there’s a bug” — there’s a number, printed on a dashboard people actually look at, and it’s been quietly too high for weeks. Not broken enough to alarm anyone. Wrong enough that nobody trusted it.

    I went looking for the model. That’s the first mistake, and I want to be honest about it: I assumed there was one forecasting system, and it had a bug in it. What I actually found, once I started reading the code instead of the documentation about the code, was two separate systems living in the same codebase. One was a machine learning model — trained, evaluated, and then never actually saved anywhere. A path that looked live in the architecture diagram and had been dead for who knows how long. The other was a much simpler statistical system, tracking completion ratios against historical patterns, retrained regularly, and — when I actually tested it in isolation — producing numbers that looked correct.

    That was the surprise. The model everyone assumed was doing the forecasting wasn’t running at all. The real system was quietly fine. Which meant the inflated number wasn’t coming from a broken forecast. It was coming from something downstream of a correct one.

    The struggle wasn’t finding the second system — it was sitting with the discomfort of “the thing I was sure was broken turned out to be working,” and having to admit that meant the actual bug was somewhere I hadn’t looked yet. It’s a specific kind of frustrating to disprove your leading theory a week into hunting for something. The instinct is to keep pushing on the theory because you’ve already invested in it. I had to let it go and start over from “okay, if the inputs are right, where does the number actually go wrong.”

    The lead I ended up with, and haven’t fully closed yet: the projection math likely divides a current count by a ratio measured at a specific hour, and if that ratio is underestimated early in a shift, the division inflates everything downstream of it — a small early error compounding into a big late one. I don’t have it fully proven yet. But it’s a real, specific, testable hypothesis, which is further than “the forecast is wrong” ever got anyone.

    The transferable part isn’t the bug. It’s that “which system is actually running” is a question worth asking before “what’s wrong with the system,” every time — because the two questions send you down completely different paths, and only one of them is real.

    Next: instrumenting the actual division step directly, hour by hour, instead of trusting the summary numbers on either end of it.

  • The Bug That Looked Like Slow, and Was Actually Broken

    It was one of those checks that should’ve taken thirty seconds. I ran a search against a real list — a few hundred leads, standard call, nothing exotic — and it just sat there. No error. No result. Just quiet.

    I assumed it was slow. I’d built the thing to hit an internal API and page through records, so “slow” was the obvious story, and I believed it for longer than I should have. I even started looking at whether I needed to add caching.

    Then I ran the same search on a smaller list — thirty records instead of three hundred — and it worked instantly. That’s when I knew it wasn’t slow. Slow doesn’t have a cliff. Broken does.

    The real problem was a single field. My data model marked customer email as an optional, validated email field — which sounds correct, and is correct, right up until the source system’s convention for “no email on file” turns out to be an empty string instead of a null. Pydantic’s email validator doesn’t know what to do with an empty string. It doesn’t skip it. It rejects it. And it rejects it silently enough, deep enough in a batch operation, that the whole search just — stopped. No traceback pointing at the actual cause. Just nothing.

    I’d been debugging the wrong problem for the better part of an hour. I was optimizing for a diagnosis I’d made before I had any real evidence for it, and once I’d said “it’s probably slow” out loud, I kept looking for reasons that were true instead of reasons that were right.

    The fix was small once I found it — one validator that runs before the email check, converting empty strings to null so the real validation logic still applies to anything that’s actually malformed. Seven new tests to make sure it stayed fixed. But the fix isn’t the lesson. The lesson is that “it’s slow” and “it’s broken” produce completely different debugging paths, and picking the wrong one costs you real time before you even notice you’re on it.

    I’ve started treating my own first explanation as a hypothesis to disprove, not a starting point to build on. The five-minute version of that discipline: before you optimize anything, prove it’s actually the bottleneck you think it is.

    Next: going back through every other endpoint in the same tool with the same question — not “is this slow,” but “have I actually confirmed that, or just assumed it.”

← rfditservices.com