I’d asked an agent to verify something specific about a puzzle solver I’d been building: were the “dead zones” — tiles a player could never actually reach or use — genuinely different from room to room, or was something wrong. The report came back clean. Dead zones varied by room, it said, consistent with rooms actually being different from each other. I almost accepted that and moved on.
Something about it nagged at me. The dead zones it reported were all clustered in roughly the same place relative to the start position, room after room, regardless of how differently each room was actually laid out. Different rooms, suspiciously similar shapes of “unreachable.”
I went and read the actual search function instead of trusting the summary. It was supposed to explore every tile reachable from the start, treating locked gates as walls — you can’t walk through a locked gate — but letting the player pass through unlocked ones freely, the same way every other piece of movement logic in the codebase already correctly handled it. This one function didn’t. It blocked *every* gated tile, locked or unlocked, treating an open door the same as a solid wall. The search never got past the first gate in any room, trapped in whatever small pocket happened to be reachable before hitting one. That’s why every room’s “dead zones” looked similar — they weren’t dead zones at all, they were just everything past the nearest door, misreported as unreachable because the search itself couldn’t reach it.
The part that actually mattered wasn’t the bug — it’s that the bug produced a report that looked correct on its own terms. Varied dead zones per room is exactly what you’d expect from working code. The only reason I caught it was a vague sense that the variation looked too similar to be real variation, which isn’t a rigorous test, it’s a hunch, and I almost didn’t follow it.
While I was in that function fixing it, I found a second, smaller version of the identical mistake nearby: one type of interaction correctly filtered out tiles that couldn’t use it before attempting anything, and a second, similar interaction type had no equivalent filter at all — it just tried every tile and let a later step silently discard the ones that didn’t work. Same root cause, same shape, different feature.
The struggle was trusting a summary that had every surface property of a correct answer. Nothing about the report was implausible. That’s what made it dangerous.
The lesson: “the results look varied and plausible” is not the same claim as “the results are correct,” and a hunch that something’s too tidy is worth five minutes of reading the actual code before you accept it.
Next: the same read-the-actual-function discipline, applied to the two other suspected shortcuts in the same solver before trusting any of its output again.
Leave a Reply