There was a rule that had been running for a long time: if a list’s contact rate drops below a fixed number after enough attempts, cut it and move to something else. Simple, defensible-sounding, the kind of rule nobody questions because it’s been there since before anyone currently on the team arrived.
I sat down to actually check what that rule was defensible *against* — not what it claimed, what it proved. A flat cutoff at a fixed percentage doesn’t account for how much you can trust an observation at a given sample size. Cut a list at 400 attempts and 0.25%, and the honest, statistically rigorous floor that observation actually clears — accounting for the real uncertainty at that sample size — turns out to be 0.056%. Not 0.25%. Seven times lower. The rule’s name promised one standard and delivered a much more trigger-happy one, and nobody could see the gap because nobody had run the number.
That explained something that had been bothering people for longer than the rule itself had existed: cuts kept getting reversed. A list would get pulled for underperforming, and later turn out to have been fine. I went and checked, against the real history, how often that happened — pulled every moment the rule would have fired across a quarter of real data, then checked what those same lists did in the following day. About one in six to one in four of them recovered on their own within 24 hours. Not because the rule was wrong to exist. Because it had never been calibrated to know the difference between “actually bad” and “noisy this hour.”
While I was in that part of the system, I went looking for the audit trail — the log of every automated decision the balancer had ever made, expecting to be able to reconstruct exactly which lists got swapped for what reason. The log existed. It had the right columns for it — which list got removed, what replaced it. Every single row had those columns empty. Every reason field said the same generic string, verbatim, on all thirty-four thousand rows. The audit trail had been built and never actually wired up to record anything real.
The struggle in both of these wasn’t technical — it was resisting the instinct to fix the surface symptom (adjust the cutoff number) instead of asking whether the whole shape of the rule was the problem. A better number on a badly-shaped rule is still a badly-shaped rule.
The lesson: when a threshold has a specific number in its name, ask what that number is actually defensible against, not what it claims to be. The two are not always the same thing, and the gap between them is where false confidence lives.
Next: replacing the flat cutoff with a rule that adjusts automatically for how much history a list has behind it, instead of hand-picking a new number every time the old one stops working.