It was 2:46 on a Sunday afternoon when I asked Claude if it wanted to take a look at the codebase. Trailswarm had been public for two days.
It started by counting. Since Thursday, main had taken on 852 commits. Pull requests 162 through 172, eleven of them, had all merged inside about seventeen hours. I asked to see the newest forty commits. Twenty-seven were the queue moving a directive file from one state to the next. Five were merges. Eight were code or docs: a Mario Land search tweak, a map view fix, a corrected line in the results file.
Then the numbers that mattered. The checker I use to tell whether the first goal is done had found one unbroken run from the bedroom to a badge: 8,003 presses, zero Claude consults. It also reported that 1,483 of those presses were tagged human. That is 18.5 percent of an “unbroken” run.
Then the test suite. The merge gate runs the tests in a clean copy of the repo, with no ROM and no local data. In that copy the suite had shown zero failures and 40 skipped tests. When Claude ran the same suite on my machine, where the ROM and the data exist, it took three minutes and forty-five seconds and finished with 5 failed, 2,184 passed, 1 skipped. Four of the failures were in the behavior that reorders my Pokémon party. The gate had been green the whole time. It just never watched that code run against real data.
At seven that night I told Claude the thing I’d been circling: “The project remained interesting, it’s more it stopped making recognizable or interesting progress, it came down to polish, and extending proved to be more work.”
Looking back at the afternoon, the mistake is easy to name. I’d been counting the wrong things. Commits, merged pull requests, a test count with a blind spot, and a done-condition that couldn’t tell the agent’s presses from a human’s. Every number was real. None of them was what I actually wanted, which was to look at the swarm and recognize that it had gotten better.
Progress I can’t recognize isn’t progress to the person who has to keep caring about it, and a green gate that can’t see real data isn’t a gate.
That night nobody knew whether the tests or the code were wrong. The process fix was clear either way: run the suite twice before anything merges, once in a clean copy and once with the ROM and local data present. If the code turns out to be wrong, real runs have been affected since pull request 166, and I’ll say so in a follow-up. The code is public: Trailswarm on GitHub.
