Robert Dugger — Dev Blog

Tag: python

  • I Processed 671,000 Records in 6 Minutes and 32 Seconds

    The number that mattered wasn’t 671,000. It was 6:32.

    But before I got there, I had to survive a BOM file.

    What a BOM File Does to Your Morning

    A BOM — Byte Order Mark — is a hidden character. Three invisible bytes at the start of a UTF-8 encoded file, placed there by certain export tools as a signature. Completely benign in most contexts. Catastrophic if you’re parsing column headers programmatically and nobody told you it was there.

    The file came in as a standard monthly lead drop from a third-party vendor. CSV, normal structure, expected columns. I loaded it, ran my process, and watched it fail in a way that made no sense. The column I was looking for was right there in the header row. My code couldn’t find it.

    I opened the file in a hex editor. The first column name didn’t start with the letter I was looking at. It started with EF BB BF followed by the letter. Three bytes of invisible garbage prepended to the header, making Name into something my string comparison had never seen before and would never match.

    That was lesson one: files lie. Specifically, files produced by systems you don’t control lie in ways you won’t anticipate until they do it to you. The fix was one line. The lesson was architectural.

    Tool One: The Fuzzy Scrubber

    The lead data problem predates the BOM file. It starts with a simpler, more persistent irritant: duplicate companies.

    When you’re working with raw lead data at scale — scraped data, third-party drops, list purchases — you consistently encounter the same fundamental problem. A regional franchise has fifty locations. A corporate chain has a hundred. A national company has branch offices in every market you’re targeting. Each one appears as a separate row with a slightly different name. Smith Plumbing, Smith Plumbing LLC, Smith Plumbing of South Florida, Smith Plumbing — Boca Raton.

    You don’t want fifty versions of the same company. You want one, or none.

    The first tool I built was a fuzzy scrubber. Not exact match deduplication — exact match is easy and catches almost nothing. Fuzzy matching: similar names, above a threshold, clustered and collapsed. The goal was to identify companies that were likely regional, corporate, or franchise operations and remove them from the working set before they reached the dialer.

    The first threshold caught seven clusters. Too aggressive — legitimate distinct companies were getting collapsed. I tuned it. Three clusters. Better. Still not perfect, but better is the goal in data work. Perfect is a fiction that costs you the pipeline.

    The scrubber became step one of what would eventually be a twelve-step process. I didn’t know that yet.

    The Encoding Problem

    Every data engineer eventually learns that text encoding is not a solved problem.

    It’s solved in theory. UTF-8 is the standard. Everyone agreed. The agreement doesn’t survive contact with files produced by legacy systems, Windows-default exports, Excel users who have never thought about encoding in their lives, or third-party vendors whose ETL tools were written in 2003.

    The lead data came from multiple sources. Each source had its own encoding habits. Most of the time UTF-8 worked. Sometimes it didn’t, and the failure mode wasn’t a clean error — it was silent corruption. Characters mangled into question marks or replacement symbols. Phone numbers with invisible characters that made them unparseable. Company names with encoding artifacts that defeated fuzzy matching and left junk in the dataset.

    The encoding handler became step two. Detect the encoding before you process. Normalize to UTF-8 explicitly. Validate that the result is clean. Only then proceed.

    The BOM problem was a subcase of this. A BOM-aware reader handles it automatically. I had not been using a BOM-aware reader. I was, after the BOM incident.

    Common Columns and the First Real Pattern

    By the time I had a fuzzy scrubber and an encoding handler, I was starting to see a pattern in what the data needed across different downstream destinations.

    We run multiple dialer systems. Each dialer has its own expected column format. The CRM backend has its own schema. A lead that’s processed correctly for one system is formatted wrong for another. If you’re loading data manually into each system, this is an annoyance — you reformat before each import. If you’re trying to automate the flow, it’s a structural blocker.

    I started mapping what each system actually needed. What columns. What names. What formats. What was optional, what was required, what would cause a silent failure if missing versus an explicit error.

    The overlap was significant. Most of what dialers need from a lead record is the same: company name, contact name, phone number, address, state, some kind of category or industry tag. The differences were in naming conventions and field formats, not in the underlying information.

    This observation led directly to the most important structural decision in the whole pipeline.

    Golden Columns

    If multiple downstream systems all need roughly the same information, and the variation is in format rather than content, then there exists a canonical representation of a lead record that can be transformed into any downstream format without data loss.

    I called this the Golden Columns.

    The Golden Column set was the formal expected schema that a lead record had to conform to before it could go anywhere. Not the format any one system needed — the superset of everything any system might need, normalized to a single consistent representation. Once a record was in Golden Column format, outputting it for any dialer or the CRM was a transform, not a reconstruction.

    This was the moment the project stopped being a collection of data-cleaning scripts and started being a pipeline.

    A pipeline needs a contract. The contract defines what goes in, what comes out, and what the shape of the data is at each stage. Before the Golden Columns, I had tools. After them, I had stages. That’s a different thing. Tools are independent. Stages are composable. You can chain stages. You can add a stage without breaking the others. You can test a stage in isolation.

    The pipeline design followed from the contract almost automatically.

    Twelve Steps

    By the time I had formalized the Golden Columns, I could see the full shape of what the pipeline needed to do to take raw third-party lead data to a dialer-ready output. I wrote it out as an ordered sequence:

    Encoding detection and normalization. BOM handling. Field presence validation against the Golden Column set. Company name fuzzy deduplication. Phone number parsing and format normalization. Address standardization. State code normalization. Industry and category tagging where present. Missing field handling and defaults. Golden Column output generation. Dialer-specific format transforms. Final validation pass.

    Twelve steps. Each one a discrete, testable operation. Each one necessary. Each one the result of a specific failure or discovery from the months of one-off processing that came before.

    671,000 records. Six minutes and thirty-two seconds.

    The speed came from the architecture. When every step is a discrete operation on a structured dataset — not a row-by-row loop, not a nested conditional mess, but a vectorized operation on a typed frame — the performance is a consequence of the design, not a separate optimization pass. I profiled it anyway. The bottleneck was where I expected it: the fuzzy matching at scale. I gave it more room to work in batch rather than iterating. The number dropped.

    6:32. That became the baseline.

    The Lesson That Took Twelve Steps to Learn

    I didn’t design this pipeline. I discovered it.

    Every tool in it started as a one-off fix for a specific problem I hadn’t anticipated. The fuzzy scrubber came from the franchise duplicate problem. The encoding handler came from the corruption problem. The Golden Columns came from the multi-system formatting problem. The BOM handler came from a hex editor at 9am wondering why a column name that was clearly visible was unreadable by my parser.

    None of it was planned. All of it was necessary.

    That’s how real data infrastructure gets built in practice: not from a design document, but from an accumulation of problems that eventually reveal the shape of the system underneath them. The design document comes after, when you’ve seen enough of the problems to know what the system is actually doing.

    The danger is stopping before you write the design document. If I’d kept the twelve steps as twelve separate scripts, I’d have twelve places to maintain, twelve places to break, twelve things to run in the right order from memory. The pipeline consolidates that into one process with a contract.

    The Golden Columns are that contract. Once you have a formal expected schema for your data, you have something you can build a tool around instead of continuing to improvise around a problem.

    The twelve steps were the improvisation. The pipeline was the design.

    The Processes Aren’t Proprietary. The Problems Are Universal.

    These patterns aren’t specific to one system or one company. They’re the natural result of working with third-party lead data at scale, and the order you build them in will follow the same logic regardless of your stack or your source.

    The fuzzy deduplication problem exists everywhere franchise and regional data gets aggregated. The encoding problem exists everywhere data crosses system boundaries. The multi-system schema problem exists everywhere more than one downstream consumer needs the same upstream data in a different format.

    The specific implementation I built is tuned to a specific set of systems, specific dialer configurations, specific CRM expectations. But the architecture transfers: identify your downstream schemas, define your canonical representation, build each cleaning step as a discrete testable stage, measure the output.

    The twelve steps I landed on were the twelve problems I encountered. Your twelve steps will be different. But you’ll encounter the BOM file. You’ll encounter the franchise duplicate problem. You’ll hit the moment where two systems need the same record formatted two different ways and you realize you’ve been solving the wrong problem.

    When you get there, you need a contract. Define what a clean record looks like before you worry about what any specific system needs from it. Everything else follows from that.

  • From Pong AI to Play Store: How a Childhood Hobby Became a Rust Game Engine

    The first game I wrote with any real ambition wasn’t a game. It was a NEAT implementation that learned to play Pong. I wasn’t trying to ship anything — I was trying to understand how a system could learn to do something I taught it.

    That question has been running in the background of everything I’ve built since.

    TurboShells came next. I took the NEAT studies from PyPong and asked: what if instead of teaching an AI to hit a ball, I bred turtles? Each turtle’s body was drawn entirely from its genes — no sprites, pure math. Shell radius, leg length, color — all expressed from a genetic sequence at render time. They raced. The faster ones bred. The slower ones didn’t.

    Nobody played TurboShells. But I learned something: the genetics loop — dispatch a breeding pair, wait for the outcome, observe the consequence — was more interesting to me than any game mechanic I’d seen. I wasn’t building a racing sim. I was building a system that made things happen without me.

    ChimeraLab was the first time I tried to give the genetics a body.

    Custom 3D creature rendering. SpineComponents — oblongs stacked together, body parts articulated from code. I got a humanoid assembled. I gave it a skeleton. I ran the simulation.

    It fell to the floor under its own weight.

    I never did fix the bipedal problem. But I got it to transition from two legs to four using a slider, and watching that happen — a creature renegotiating its relationship with gravity in real time — taught me more about 3D rendering than any tutorial I’d read. Sometimes the failure is the lesson.

    rpgCore was the foundation I should have built first. A thousand tests. Real ECS architecture. Genetics, lifecycle, dispatch — everything composable, everything verified. SlimeGarden put it to work: an astronaut crash-lands on an unusual planet and finds slimes. Breed them. Dispatch them. See what comes back.

    It sounds simple. It wasn’t. And it pointed somewhere.

    Seven projects in, I was staring at the Google Play Store submission checklist.

    OperatorGame ran on Android. Real Rust, real Bevy, real APK on a real phone. The combat worked. The UI was clean. I’d solved the hard problems.

    The submission required a 512×512 app icon, a 1024×500 feature graphic, and two screenshots.

    I didn’t have any of them.

    I could have made them. It would have taken an afternoon. But sitting there looking at that checklist, I realized the assets weren’t the problem.

    The problem was I had no audience. I was about to pay the Play Store’s attention tax — discoverability weighted toward downloads, downloads toward reviews, reviews toward players who found you somehow — with zero players behind me. I wasn’t Android Store money-ready. I was Itch.io audience-ready.

    Those are different things. Confusing them is expensive.

    Here’s what the journey looked like from the inside:

    PyPong AI taught me how systems learn. TurboShells taught me that genetics loops are more interesting than game mechanics. ChimeraLab taught me that creatures fall down and that’s instructive. rpgCore gave me the foundation. SlimeGarden gave the foundation a story. OperatorGame proved the Android pipeline. VoidDrift took the dispatch loop — Scout mines ore, returns, consequence — and dressed it in something people want to watch.

    Every project is the same loop. Something dispatches. It does its work. It returns with a result. Something changes.

    I’ve been building that loop for years. I just didn’t see it until I looked at the list.

    The lesson isn’t “don’t aim for the Play Store.”

    It’s: know what you’re ready for. The Play Store is a distribution problem you solve after you have players, not before. Itch.io is where you find out if anyone cares. If they do, the Play Store is a next step. If they don’t, you learned that cheaply instead of expensively.

    VoidDrift is on Itch right now. A small audience that keeps coming back. That’s the signal I was missing with OperatorGame.

    When the audience is real, the Play Store assets take an afternoon.

    There’s a story that keeps circling my mind. Someone in a ship, traveling through a black hole, becoming something else. They find a station. What follows is a macabre exploration of self — what survived the transit, what didn’t, what the new thing is capable of.

    VoidDrift is the ship and the void. SlimeGarden is the crash-landing after.

    The loop doesn’t end at the Play Store. It ends when the story does.

    The Scout dispatch loop in VoidDrift is the same loop TurboShells was running — breed, wait, observe — dressed in space mining clothes, nine projects later. Phase 5 is live. The Play Store listing is three PNGs away.

    The story is still circling. I’m still building toward it.

  • I Built a CLI to Replace Expensive AI Directive Generation

    I Built a CLI to Replace Expensive AI Directive Generation

    The friction point is simple to describe. Claude designs the architecture. Windsurf builds it. The directive that connects them — structured, scoped, phase-gated — gets written by me, by hand, every single time.

    I’m the middleware. I built a tool to replace myself. It didn’t quite work.

    OpenAgent started from a real observation: the same context was being re-explained in every session. I had architectural patterns, stop rules, test floor conventions — and every new Windsurf session, the agent had no idea any of it existed. The directive was the missing connective tissue. Write it well and the agent stays on scope. Write it badly and the agent invents its own architecture.

    So I built a CLI that reads the codebase, understands the structure, and generates directives shaped to my development style. The breakthrough was SOUL.md — eight questions about how I actually work. That profile gets embedded in every directive. When OpenAgent generates something, it references the right conventions, names the right stop conditions. It sounds like something I’d write.

    It’s on PyPI as openagent-directive. v0.2.2. 103 passing tests.

    Here’s the part I didn’t put in the README: I still do the same manual cycle.

    The tool works. The directives it generates are useful. But I’m still the one handing them to Windsurf, watching the session, course-correcting when it goes sideways. The friction I wanted to eliminate is still there because the real blocker isn’t the directive — it’s that there’s no coding agent that lives outside an IDE.

    Windsurf, Cursor, Copilot — they all require a human in the seat. The autonomous loop I wanted, where OpenAgent feeds a directive to a coding agent that executes independently, reports back, and waits for the next one, doesn’t exist yet in any reliable form. The IDE-bound constraint kills the automation before it starts.

    I built a correct solution to the wrong layer of the problem.

    The pivot I keep thinking about: OpenAgent as an MCP tool. Not a CLI that generates directives for humans to hand off, but a codebase intelligence layer that a coding agent can query directly. What files are in scope? What’s the test floor? What patterns does this codebase use? An agent with access to that context doesn’t need a human to write the directive — it can construct its own.

    That version of OpenAgent is waiting on the ecosystem. When a capable coding agent exists that can operate outside an IDE, receive a structured task, execute against a real codebase, and return proof — OpenAgent is already positioned to be the interpreter it needs.

    For now it’s a CLI on PyPI that saves me twenty minutes per directive and reminds me that some problems can’t be fully solved until the infrastructure around them catches up.

    The friction is still there. The tool is ready when it isn’t.

  • What Ants Taught Me About Systems Design

    What Ants Taught Me About Systems Design

    Before programming, there were ants.

    Not metaphorical ones. Real colonies — pheromone trails that appear from nothing, queen mortality absorbed without panic, chambers organized by workers who’ve never been given instructions. No central controller. No plan. Just rules simple enough to follow and complex enough to produce something alive.

    When I started building software, I was chasing the same thing.

    AntSim is an evolutionary ant colony simulation with split-view rendering. The top two-thirds is the foraging ground — pheromone trails building and fading in real time. The bottom third is the colony cross-section: chambers, tunnels, brood, storage.

    The genetics system controls five traits per ant: sensitivity, speed, boldness, lifespan, energy efficiency. That’s it. Five genes. The colony doesn’t know it’s evolving. Individual ants don’t know they’re part of a system. They follow their traits and the colony either survives or it doesn’t.

    When a queen dies — randomly, the way queens do — workers autonomously identify royal jelly candidates and raise a successor. Colony continuity without external control. The system just continues.

    The original genetics system had too many genes. Forty variables, every trait interacting with every other. The ants evolved into creatures that couldn’t survive their own complexity — biologically sophisticated and functionally useless.

    Stripping back to five traits was uncomfortable the way all simplification is uncomfortable. It felt like giving up. What I got instead was emergence — behaviors I didn’t design, produced by interactions I didn’t anticipate, between five simple numbers and enough time.

    The cross-section visualization is still in progress. Phase 3 rendering complexity exploded the moment I made it detailed. Individual chambers, tunnel geometry, precise brood locations — the simulation could handle it but the renderer couldn’t. I scaled back. It’s still on the list.

    Here’s what I didn’t understand when I started: I wasn’t building a simulation. I was building a philosophy.

    Every system I’ve built since has the same underlying shape. Something dispatches — an ant, a Scout drone, a queued job, a breeding pair. It does its work without supervision. It returns with a result. The colony, the station, the codebase, the task queue — they change accordingly.

    That loop appeared first in an ant colony. It’s in VoidDrift’s Scout dispatch. It’s in PrivyBot’s job queue. It’s in rpgCore’s genetics engine. I’ve been building the same system for years in different materials.

    Emergent behavior doesn’t require complex rules. It requires simple constraints, honest consequences, and enough time to surprise you.

    The ants figured that out long before I did.

  • The Engine Legacy: From Asteroids to rpgCore

    The Engine Legacy: From Asteroids to rpgCore

    At some point I stopped counting how many times I’d written the same movement system.

    It wasn’t any one project’s fault. Every game had its own physics, its own collision logic, its own state management — each a reasonable decision in isolation, collectively a pattern I was tired of repeating. The tenth time you write an entity update loop you start asking a different question: not “how do I build this game” but “what would make the next one easier to start.”

    rpgCore started as a refactoring project and became something else entirely.

    The idea was simple: extract the patterns that kept repeating and build a reusable toolkit. A simplified Entity-Component-System in Python — modular enough that adding a genetics system didn’t require touching the movement code, adding a lifecycle system didn’t require touching the genetics code. Each piece independent, each piece composable.

    Simple ideas scale badly. The toolkit grew. A thousand tests. Then eleven hundred. GeneticsComponent. LifecycleComponent. DispatchSystem. ResourceFlow. A scene system, a UI theme, a demo registry. At some point it stopped being a toolkit and started being a codebase worth protecting.

    The mistake I kept almost making was treating rpgCore as a means to an end. It isn’t. The foundation is the thing.

    When you build something generic enough to support any idea, you start seeing which ideas are worth having. The genetics system in rpgCore is the same system that ran TurboShells’ turtle breeding. The dispatch pattern is the same one VoidDrift’s Scout drones use. The resource flow is the same loop that appears in SlimeGarden. rpgCore didn’t just save me from rewriting code — it made the pattern visible, and once the pattern is visible you can apply it deliberately.

    The 1,122 tests aren’t there because I’m disciplined. They’re there because a foundation without verification isn’t a foundation, it’s a guess.

    The BreedingSystem is still queued. Allele resolution, trait inheritance, phenotype expression from genotype. When it ships it connects to every genetics-adjacent idea I’ve had since TurboShells. ConquestSystem is queued behind it.

    rpgCore doesn’t ship. It makes shipping possible.

    That’s a slower kind of value than a finished game. It’s also the only kind that compounds.

  • The Hybrid Engine: Rust Performance, Python Agility

    The Hybrid Engine: Rust Performance, Python Agility

    The problem with DeFi trading bots is speed. The problem with fast code is that it’s expensive to change.

    A pure-Rust bot wins the race to the block — compiled, deterministic, fast. When the market shifts and your strategy needs to change, you recompile. Overnight. While your edge evaporates.

    A pure-Python bot iterates in minutes. It also loses to anything compiled. In a system where the difference between capturing an arbitrage and missing it is milliseconds, interpreted code is a structural disadvantage.

    I needed both. So I built a bridge.

    The hybrid architecture splits responsibility at the right seam. The Rust core handles everything where latency matters: WebSocket connections, memory-safe transaction signing, packet serialization. Compiled, stable, rarely touched. The Python strategy layer sits above it, communicating through a lightweight interface. When the trading logic changes — when a pattern emerges, when a parameter needs tuning, when a strategy turns unprofitable — you change the Python. No recompile. The execution layer keeps running.

    Decoupling execution from intelligence meant iteration speed became unconstrained by compilation time. A new strategy at midnight, tested by 2am, discarded by morning without touching Rust.

    The bridge itself was the hard part. Any interface between two languages has a seam, and seams are where bugs live. Getting data structures consistent on both sides — ensuring what Rust serializes is exactly what Python expects — required more care than either side alone. When something went wrong, it could be Rust, Python, or the interface between them. You learn to test both sides independently before trusting the combination.

    PhantomArbiter ran 400 live trades on Solana in 2025. The architecture worked. The margin didn’t scale — the arbitrage windows were narrower in practice than in theory, and at volume the economics didn’t justify the infrastructure.

    But the pattern was correct. Compile what doesn’t change. Script what does. The intelligence layer should be easy to replace. The execution layer should be hard to break.

    That principle didn’t stay in trading. It’s in every complex system I’ve built since — MCP tools handle execution, the model handles intelligence, and the interface between them is where the design lives.

    I didn’t keep trading. I kept the pattern.

  • The Insight Lens: Building Chrome Tools for Contact Center Teams

    The Insight Lens: Building Chrome Tools for Contact Center Teams

    Most workplace software was built for someone else’s workflow.

    I work in high-volume digital marketing. The tools we use — Convoso, Telesero, various CRMs — are built for general audiences, which means they’re never quite right for any specific team’s actual process. Managers get report portals that are hard to read and impossible to print. Agents get interfaces with small frictions that compound across hundreds of calls a day. Scheduling runs on whatever the vendor decided was sufficient.

    So I build tools.

    The first one came from a simple request: a manager on the scheduling side needed to present agent performance reports to stakeholders, and the vendor portal printed badly. I built a Chrome extension that reformats the output into something clean and readable. The vendor’s UI is still clunky. The reports look professional now.

    The page refresher was even simpler. The team needed an auto-refresh tool. The right answer wasn’t a third-party extension — those tools read browser data, and I wasn’t willing to risk company information for convenience. The private version took ten minutes to build and doesn’t touch anything it doesn’t need to.

    The dialing tools are the ones that add up. An agent on Telesero or Convoso might handle 150 to 400 calls in a day. Shaving three seconds off each one recovers somewhere between seven and twenty minutes per agent per day. I’ve built several of these — small quality-of-life upgrades that make each dial slightly faster, each disposition slightly cleaner. Across a team, the math compounds.

    The ambitious one was a remote coaching tool — filling a gap between what the two vendor platforms offered separately and what the team actually needed when working them together. Still in progress.

    The lesson isn’t about Chrome Extensions. It’s about proximity. Being close enough to the actual work to see the friction, and skilled enough to do something about it. That combination — operational context plus technical ability — is rarer than either one alone.

  • Why I Put a Genetic System in a Turtle Racing Game

    Why I Put a Genetic System in a Turtle Racing Game

    The NEAT algorithm that taught a paddle to play Pong is the same algorithm that maps genetic traits. Someone pointed that out and I couldn’t stop thinking about it.

    TurboShells started as a question: what if the turtle’s body came from its genome? Not as an abstraction — literally. The genome string B1-S2-P0-CFF0000 encodes four values: body type, shell type, pattern, color. The frontend parses that string and assembles the turtle from layered sprites with dynamic tinting. Change the genome, change the animal. Every turtle on the roster looks different because every turtle is different — genetically.

    That’s the Paper Doll system. A compact encoding that drives visual rendering, persists across races, and accumulates history. A turtle that wins races has a record. It has traits. It has a string that says exactly what it is.

    The project grew. What started as a breeding simulation became a real-time multiplayer racing game: FastAPI backend running 60Hz physics, WebSocket broadcasting race state at 30Hz, React and PixiJS on the front end interpolating between ticks. An NPC Manager generates persistent AI turtles that populate the roster when human players aren’t racing.

    The architecture is genuinely layered — simulation core, server bridge, frontend rendering each in their own boundary. The genome doesn’t know about rendering. The race engine doesn’t know about WebSockets. The Paper Doll assembler doesn’t know about physics.

    The lesson from PyPong was that the interesting thing about NEAT wasn’t the Pong — it was the emergence. Random variation, selection pressure, something that looks like intelligence appearing from simple rules.

    TurboShells is the same idea with different materials. The turtle’s genome isn’t optimized by NEAT weights. It’s expressed as a visible body, a race record, a persistent identity. You’re not watching neural networks compete — you’re watching genetic variation play out in real time across a roster of animals that have history.

    That’s the loop I keep building. Different game, different encoding, same underlying question.

  • Teaching Pong to Play Itself: My First Neural Network Experiment

    Teaching Pong to Play Itself: My First Neural Network Experiment

    Pong is the right choice for a first experiment because it has almost no variables. Two paddles. One ball. If you can’t teach an AI to play Pong, you can’t teach an AI anything.

    I used NEAT — NeuroEvolution of Augmenting Topologies. It doesn’t just adjust weights on a fixed network structure. It evolves the topology itself, starting minimal and adding complexity only when it helps. The training runs headless at 500x real-time speed; a separate visual mode exists purely to verify that what trained actually works. Generation 0: random paddle movement, 0% win rate. Generation 50: 98% win rate, predictive tracking.

    The difference between reacting and anticipating is memory. Standard feedforward networks see the current frame. Recurrent Neural Networks carry memory of previous states — ball velocity, trajectory history. That’s what gives the Gen 50 agent its characteristic quality: it moves to where the ball will be, not where it is. The RNN is what upgrades NEAT from “learns to respond” to “learns to predict.”

    The first training approach was pure ELO. Score points, survive, reproduce. The population converged fast — too fast. By generation 20, every agent played the same way. Safe returns, center positioning. They’d found a local maximum and stopped. No one was discovering anything.

    Novelty search fixed it. Instead of rewarding only performance, you reward uniqueness — points for behaviors the population hasn’t tried. The diversity pressure kept agents exploring. Agents with strange positioning, unusual angles, aggressive strategies started appearing — and some of them turned out to be genuinely superior. The “wrong” strategy was actually better. Pure optimization would never have found it.

    Any system without diversity pressure converges on the same answer. It finds the local maximum and calls it done. That lesson applies well beyond neural networks.

    What didn’t work: high mutation rates to accelerate training. The population collapsed — agents changed faster than they could build on what worked. Every generation erased what the previous one had learned. Slowing it down made the evolution meaningful. Some processes can’t be accelerated without destroying the thing that makes them work.

    This was the first project. Everything since has the same shape: variation, selection, emergence you didn’t design. TurboShells encoded the same loop into turtle genetics. rpgCore formalized it into a composable system. VoidDrift runs it as a drone dispatch loop.

    The Pong agent that discovered a non-obvious return angle at generation 47 is the ancestor of all of it. I just didn’t know that yet.

  • Solana Arbitrage: What I Learned From 400 Trades (And $4 in Losses)

    Solana Arbitrage: What I Learned From 400 Trades (And $4 in Losses)

    I built PhantomArbiter to detect and execute arbitrage on Solana. After 400 live trades across 3 months, I lost $4. Here’s what went wrong — and why the technology actually worked.

    The Setup

    Detect price divergence across Solana DEXes (Jupiter, Raydium, Orca, Meteora). Execute buy-low / sell-high atomically via JITO bundles for MEV protection.

    $500 initial capital. Real money. 3 months. 400 completed trades. Net result: -$4.23.

    Why It Failed

    RPC Latency

    Solana blocks come every 400ms. The system detects an opportunity, but by the time the bundle submits, 2-3 blocks have passed. The spread that looked profitable is now break-even or negative.

    Local detection: 10ms. RPC call: 50ms. Signature submission: 100ms. Next block: 400ms. Too slow.

    Professional MEV bots use validator infrastructure — direct connections, guaranteed inclusion. I used public RPC. Not competitive.

    Network Congestion

    Solana’s network is unpredictable. Sometimes transactions confirm in 1 block. Sometimes 10. Arbitraging on 1-2% margins, network variance turns winning trades into losing ones. My math said $2.50 profit. Slippage ate it before execution landed.

    Bundle Fees

    JITO bundles cost ~0.00005 SOL per transaction — $0.002-0.004 per trade. 400 trades, ~$1-1.50 in fees. The average arbitrage spread before costs was around $1.50. After slippage, fees, and MEV tax, nothing was left.

    Why It Actually Worked

    The system architecture was sound. 400 trades without a crash:

    Zero transaction failures. Zero contract bugs. Zero memory leaks. Stable WebSocket price feeds for 24/7 uptime.

    The software worked perfectly. The economic model didn’t. That’s an important distinction.

    Arbitrage at retail scale on Solana isn’t viable right now — not because the code is broken, but because professional operations have better infrastructure, lower fees, larger capital to absorb slippage, and faster access to the same opportunities. The edge isn’t available at the level I was operating.

    What I Kept

    The Rust/Python hybrid architecture — Rust handling the execution layer, Python handling strategy logic — transferred into other work. The execution core doesn’t know what it’s trading. The strategy layer doesn’t know how fast the core is running. That decoupling is the right design regardless of what market it’s applied to.

    The code got archived. The pattern didn’t.

    PhantomArbiter trades live markets, handles real network conditions, survives real slippage, and loses money honestly. Most trading systems are backtested and overfitted, profitable in theory but brittle in practice, or they don’t exist at all. A system that ran 400 live trades and lost four dollars is actually a reasonable outcome. It proved what it needed to prove.

    I didn’t keep trading. I kept the blueprint.

← rfditservices.com