Robert Floyd Dugger — Writing

Tag: python

  • How to Automate YouTube Shorts with Python and FFmpeg

    A single recording session can produce a month of Shorts. OBS captures the source video; Python extracts the clips; FFmpeg assembles them with title cards and audio normalization; the results land in a scheduled queue. Here’s how the pipeline works and where the real technical problems live.

    ## The Manual Alternative

    Without automation, every Short is a separate project: import the clip, trim it, add text, normalize audio, export, upload, write the description, schedule. Multiply that by daily publishing and the content process consumes more time than the recording session that produced it.

    The pipeline inverts that ratio. Recording remains manual — you capture the session, play the game, narrate the moment. Everything after the recording is automated. A two-hour recording session produces a batch of Shorts, scheduled across the next two weeks, without touching a video editor.

    ## Pipeline Architecture

    **Stage 1: Capture.** OBS records the session with NVENC hardware encoding. The recording is a single long source file — one session, one file.

    **Stage 2: Beat extraction.** A Python script reads a YAML file that defines the beats — timestamp pairs marking where each clip starts and ends. The beats are written during or after the session, while the content is fresh. Each beat includes metadata: the clip title, any title card text, whether the outro should be appended.

    **Stage 3: Clip assembly.** For each beat, FFmpeg extracts the clip from the source file and assembles the output: title card burned in, audio normalized, outro appended if configured. The output is a single MP4 per Short, ready to upload.

    **Stage 4: Upload queue.** The assembled Shorts land in a staging directory with a manifest file — clip filename, title, description template, scheduled publish date. Scheduled upload can be handled via the YouTube Data API (requires OAuth setup) or manually via YouTube Studio using the manifest as a reference.

    ## The Beat File

    The beat file is the human-in-the-loop interface. Everything else is automated; this is where editorial judgment lives.

    “`yaml
    beats:
    – title: “the game told me what I became”
    start: “00:34:12”
    end: “00:34:34”
    title_card: “the game told me what I became”
    outro: true

    – title: “I had no reason to be nervous”
    start: “01:02:45”
    end: “01:03:03”
    title_card: “I had no reason to be nervous”
    outro: true
    “`

    The pipeline reads this file, processes each beat sequentially, and produces one output file per entry. Adding a new Short to the queue means adding a beat entry — no video editor, no manual assembly.

    ## FFmpeg Assembly

    The core FFmpeg command for each clip handles four operations simultaneously: trim to the beat window, burn in the title card, normalize audio, and concatenate the outro if configured.

    The title card burn-in uses `drawtext`. The audio normalization uses a two-pass `loudnorm` filter to hit a consistent LUFS target across all Shorts regardless of how the source was recorded. The outro is a pre-assembled MP4 concatenated via the `concat` demuxer.

    The key parameters worth tuning:

    – **Font path** — absolute path required on Windows; relative paths fail silently in some FFmpeg builds
    – **Text positioning** — pixel coordinates, not percentages; mobile-safe zones differ from desktop
    – **Audio target** — YouTube recommends -14 LUFS; clips recorded at different volumes need normalization before loudness differences affect viewer retention
    – **Output codec** — H.264 with `yuv420p` pixel format for maximum compatibility; some encoding combinations produce files that upload but play incorrectly on certain devices

    ## The Complication: FFmpeg Drawtext and Encoding Edge Cases

    The drawtext filter is where most automation pipelines stall.

    The first failure mode: drawtext silently produces a video without the title card rather than throwing an error. The font path is wrong, or the font doesn’t support a character in the title, or the filter syntax has a subtle issue — and the output looks correct until you check whether the text is actually there. The pipeline needs an explicit check after assembly, not just a check that the FFmpeg process exited cleanly.

    The second failure mode is the outro timing. Appending an outro by extending the clip to a fixed duration doesn’t work when the clip’s natural end is close to the source file boundary — FFmpeg extends into data that doesn’t exist. The correct approach: use `ffprobe` to read the actual clip duration before assembling, then clamp the outro start point to the verified clip end. A clip that crashes FFmpeg with a duration error is more informative than one that silently produces a truncated output.

    The third failure mode is encoder-specific rendering differences. NVENC hardware encoding and libx264 software encoding produce visually similar output but handle edge cases differently. Text overlays that render correctly with software encoding occasionally produce artifacts with NVENC, particularly at high character density. Testing the full pipeline with both encoders before committing to one prevents discovering this in production.

    ## Scheduled Upload

    The YouTube Data API supports programmatic upload and scheduling. OAuth2 credentials are required — the setup is a one-time process through Google Cloud Console, and the credentials are stored locally.

    The API call sets the video status to `private` with a `publishAt` timestamp in the future. YouTube schedules the publish automatically. The pipeline writes the API response (video ID, scheduled timestamp, title) to a log file that serves as the upload manifest.

    What the API doesn’t provide: thumbnail upload is a separate call, and the thumbnail must be uploaded after the video is created. If thumbnails are part of the pipeline, the upload sequence is: create video → wait for processing → upload thumbnail.

    ## What This Doesn’t Cover

    The pipeline as described handles clip extraction and assembly from a single long recording. It doesn’t handle:

    – **Multi-source assembly** — combining clips from different recording sessions into a single Short
    – **Motion graphics** — complex animated overlays require a compositor (DaVinci Resolve, After Effects) rather than FFmpeg alone
    – **Thumbnail generation** — thumbnail design is still manual; the pipeline can insert a pre-designed template but not generate one
    – **Analytics feedback loop** — the pipeline schedules content based on a manual calendar; incorporating retention data to adjust scheduling cadence requires a separate analytics layer

    ## Frequently Asked Questions

    **Does this work on Mac or Linux, or only Windows?**
    The pipeline is OS-independent. FFmpeg runs everywhere, and the Python script has no platform-specific dependencies. The font path format differs between operating systems — Windows uses backslashes, Mac and Linux use forward slashes — and the font file itself needs to exist at the specified path on each machine.

    **Can this handle vertical (9:16) and horizontal (16:9) sources?**
    Yes, with separate output configurations. The drawtext coordinates and crop parameters differ between orientations. The beat file can specify orientation per clip, and the assembly step selects the appropriate FFmpeg filter chain.

    **What’s the minimum recording quality needed?**
    The pipeline preserves source quality; it doesn’t enhance it. Audio normalization helps inconsistent microphone levels but doesn’t fix poor room acoustics. For Shorts specifically, 1080p source at 60fps gives enough headroom for the encoding pipeline without quality loss in the output.

    ## The Code

    The ContentPipeline source is on GitHub at github.com/rfd62794/ContentPipeline. The pipeline is structured as a Python package with separate modules for beat extraction, FFmpeg assembly, and upload management. It’s documented with examples and is currently in active production use for a gaming channel publishing daily Shorts.

  • A Year In, I Realized I Had Automated My Job Away

    A year into my role as a Data Administrator, I realized I had automated my job away.

    Not in a dramatic moment. No alarm going off. Just a Tuesday morning when I opened the dashboard, saw the numbers had already updated, the lists had already swapped, the DNC submissions from the night before had already been processed — and there was nothing left to do that I hadn’t already built a system to do.

    The title on my badge still said Data Administrator. The work didn’t need one anymore.

    —

    Here’s what the job looked like when I started.

    Every morning I pulled the same data by hand. Called up the dialer reports, copied the numbers into a spreadsheet, formatted the columns, sent the summary to the team. The process took forty-five minutes and produced information that was already two hours old by the time anyone read it.

    When a manager needed to remove a number from the calling lists — a DNC request, a customer who called back angry, a number flagged by compliance — they’d send me a message and I’d log in, find the number, remove it from each campaign manually. Five campaigns. One at a time. While they waited.

    When a calling list went cold — when the connect rate dropped below threshold and the system was burning through leads that weren’t converting — I’d catch it eventually, pull a fresh list, swap it in. Eventually. When I happened to check.

    The job was real work. It just wasn’t work that required me specifically. It required someone to be there, watching, responding. A warm body at a keyboard.

    I spent the first six months learning what the friction actually was. Not the friction anyone described in the job posting — the friction you find when you’re inside the operation and you start noticing which tasks happen the same way every single day, which decisions are made the same way every single time, which problems are called “just part of the job” because nobody’s questioned whether they have to be.

    I spent the next six months building.

    —

    The first thing I built was a Google Sheets dashboard that pulled the dialer data automatically. Three hundred and twenty-five summary lists. Fourteen hundred and eighty-eight end-of-day records. The morning pull that used to take forty-five minutes happened overnight without me. I opened the sheet and the numbers were already there.

    The second thing I built was a DNC tool. A Slack slash command that a manager could use to submit a number for removal. They’d type it in, hit send, and the tool would hit the Convoso API across all five active campaigns simultaneously and log the result to cloud storage. The manual process — log in, find the campaign, find the number, remove it, repeat four more times — compressed to three seconds. Twenty requests a week. Zero manual steps.

    The third thing I built was the automation that changed everything. A Python system that watched the dialer servers, monitored list performance against defined thresholds, and swapped underperforming lists automatically. Stagnation detection. Grace periods to prevent premature swaps after a reset. An autonomous loop that ran from ten in the morning to six in the evening without anyone touching it.

    By month twelve, the loop was running. The dashboard was updating. The DNC tool was fielding requests. The thing I’d been hired to monitor was monitoring itself.

    —

    The title didn’t change. The conversation about what the role had become hadn’t happened yet — that’s a different story.

    But I knew what had happened. And I knew what it meant.

    It meant the friction I’d lived inside for a year — the manual pulls, the one-at-a-time DNC removals, the list babysitting — was solvable. Not because I had special tools or a special budget. Because I was close enough to the operation to see exactly where it was leaking time, and experienced enough to build the patch.

    —

    That’s the pitch.

    Not “I know Python” — lots of people know Python. Not “I’ve heard of contact centers” — that’s worth nothing.

    The pitch is: I lived inside a contact center operation, felt exactly the friction your floor feels, and built systems that eliminated it. I know what it costs you in manager time when a DNC request sits in someone’s inbox for four hours. I know what it costs you in lead quality when a stale list runs two hours past its useful life. I know what it costs you to have a person watching a dashboard that should be watching itself.

    I built the solution for my own operation. I can build it for yours.

    If your floor has this friction, the intake form is at rfditservices.com/intake.html. The first conversation is free. The friction isn’t.

  • I Didn’t Build the Pipeline I Planned. I Built the One That Worked.

    The original idea was clean. You point AI at a mobile game, it scouts the mechanics, writes the script, generates the images, produces the review. No recording sessions. No editing. No you, really — just a prompt and a published video.

    That’s the pipeline I set out to build.

    It didn’t work.

    Not because the individual pieces were broken. The scouting ran. The scripts came back. The images generated. But the output was the kind of content that exists in a category by itself — technically complete, immediately recognizable as something a human didn’t make. Nobody wants to watch it. I didn’t want to watch it.

    The surprise wasn’t that fully-automated AI content looked bad. The surprise was how long I kept trying to fix it before accepting that this particular problem wasn’t an engineering problem.

    So the pipeline changed shape. If AI couldn’t generate the review, maybe it could at least handle the production. I play games anyway. I could record the session, feed it through OBS automation, let FFMPEG cut the interesting parts and ship them as Shorts. Less ambitious. More honest about what I was actually building.

    OBS never cooperated. The API surface, the timing, the state management — every integration attempt landed somewhere between fragile and broken. I spent sessions I didn’t have on a problem that kept reforming.

    I left OBS alone and worked with what I had: recorded footage, a transcription pipeline running Whisper on Tower, and FFMPEG.

    The FFMPEG path worked.

    Record a talk-over session. Transcription runs on Tower. FFMPEG processes it into a Short. The upload schedules. I play the game. Everything else is automated.

    That’s the pipeline that’s running now. Three weeks of building — posting the whole time — bought six weeks of runway. Six games in rotation, Shorts publishing daily, the whole thing running on Tower while I’m at the day job.

    What I understand now: the goal was never full automation. The goal was removing friction from the parts of the process that don’t require me. Playing the game requires me. Deciding what’s interesting requires me. Transcribing, cutting, formatting, uploading — none of that does.

    The pipeline I planned would have removed me from the parts that matter and kept me in the parts that don’t. The pipeline I built got that backwards and arrived at the right answer.

    The AI didn’t write the review. It wrote the caption. That turned out to be the correct job.

    The repo still has the old layers in it — the scouting code, the failed OBS integrations, three distinct shapes the project wore before landing on this one. I haven’t cleaned it up. The dead code is an honest record of what I thought this was going to be.

    The next version is probably a clean rebuild as an MCP server, now that I know what the actual tools are. Five or six tools. Transcription, processing, scheduling. Nothing it took three attempts to discover I didn’t need.

    That’s the pipeline the first version was trying to become.

  • The Pipeline That Runs While I Sleep

    I woke up on a Wednesday and checked Tower before I checked anything else.

    RALPH had fired at 2:47am. Detected a pattern in the task queue, ran a research chain, summarized the output, and filed it. No prompt from me. No session open. I was asleep. The system decided something was worth doing and did it.

    That’s the thing I built toward for two years of warrior sessions. Not the feature. Not the specific task RALPH ran that night. The fact that it happened without me.

    What’s Actually Running

    There are four systems on Tower right now that operate independently of my presence.

    PrivyBot is the oldest and most capable. It’s a personal autonomous AI assistant — Python, FastAPI, 131 MCP tools, running as an NSSM service on Tower. It has a priority queue, an async task loop, and RALPH: a persistent overseer that fires on schedule and monitors for things worth acting on. Email summaries. GitHub activity. YouTube analytics. Game metrics. It doesn’t wait for me to ask. It runs its own loop and surfaces what matters.

    The test floor is 557 passing, 0 failing. I know that number is real because I certified it myself. Every phase of PrivyBot’s development ended with that verification before the next phase started. 33 phases. The floor moved up each time, never down.

    ContentPipeline is the YouTube operation. I play games. The pipeline records the session with OBS, transcribes it with Whisper running on Tower’s GPU, identifies the moments worth keeping, generates captions, assembles the Short with FFMPEG, and schedules the upload. The calendar runs through July 2026 without me touching it. The pipeline built the calendar. I just played the games.

    TeleseroAdmin2026 runs during business hours without supervision. It watches six dialing servers, monitors list performance, and swaps underperforming lists automatically based on thresholds I defined. 262 tests. Full-auto loop. The intervention it was built to eliminate — me watching metrics and making manual swaps — hasn’t happened in months.

    DNC Automation runs on Cloud Run. Compliance checks that used to be a manual process, now a deployed service. Stable. I check it roughly every two weeks to confirm it’s still running. That’s the entirety of my interaction with it.

    How You Get There From a Warrior Session

    None of these started as systems. They started as scripts.

    PrivyBot started as a Telegram bot that could answer questions. TeleseroAdmin2026 started as a Python script named by date that logged into a portal and swapped one list. ContentPipeline started as a single produce_short.py file that required manual input at every step.

    The path from script to autonomous system is always the same and always takes longer than you expect.

    First you automate the thing you do most often. Then you notice the thing adjacent to it that you’re still doing manually. You automate that. Then you realize the two automations need to talk to each other, which requires a shared config. The shared config implies a shared schema. The shared schema implies a system.

    You don’t design the system. You discover it. The design document comes after, when you’ve accumulated enough automated pieces to see the shape of what they’re forming.

    The warrior sessions are how the pieces accumulate. Ninety minutes on a Tuesday night adds the encoding handler. Two hours on a Saturday adds the deduplication pass. A three-hour session where something clicked adds the orchestration layer that connects them. None of those sessions felt like building a system. They felt like solving the problem in front of you.

    At some point you look up and there’s a system.

    What Autonomous Actually Means

    I want to be precise about this because “autonomous” gets used loosely.

    Autonomous doesn’t mean unsupervised forever. It means the system handles the routine cases without requiring a human in the loop for each one. The edge cases still surface. The unexpected failures still need attention. The system doesn’t replace judgment — it handles volume so judgment is reserved for the things that actually need it.

    RALPH firing at 2:47am and running a research chain is autonomous. RALPH discovering a new class of task it’s never handled before and stopping to report it rather than guessing — that’s also autonomous, in a different direction. The system knows what it knows and flags what it doesn’t.

    TeleseroAdmin2026 swapping a list because a performance threshold was crossed is autonomous. TeleseroAdmin2026 encountering a portal login flow that changed after a site update and stopping the loop rather than proceeding incorrectly — still autonomous. The right behavior in an unexpected situation isn’t always to act. Sometimes it’s to stop and surface the situation.

    The systems I trust are the ones that fail loudly when they’re outside their design envelope. The ones that fail quietly — that continue operating in edge cases and produce confident, wrong output — those aren’t autonomous systems. They’re liability.

    This is the same principle I apply to coding agents. A system that tells you it succeeded when it didn’t isn’t a trustworthy system. Raw terminal output only. The floor is real or it isn’t.

    The Compounding

    Last Wednesday RALPH ran 14 tool calls before 6am. By the time I was at my desk, there was a digest waiting: yesterday’s YouTube performance, an alert on a campaign metric that drifted outside threshold, a summary of three GitHub commits I’d made the night before with notes on what each one changed.

    I didn’t ask for any of it. I configured the system to care about those things, and the system cared about them while I slept.

    That’s a different relationship with work than I had two years ago, when every piece of information about my projects required me to go get it. The information is still there. The systems go get it for me and bring back what matters.

    The compounding isn’t the time saved on any individual task. It’s the accumulation of context that’s available without friction. I sit down knowing the state of things because the systems maintained the state while I was away. The warrior sessions start from a known position instead of starting with reconnaissance.

    Why This Is the Pitch

    The consulting angle I’m building toward isn’t “I’ll automate things for you.”

    It’s “I’ll build systems that maintain themselves.”

    There’s a specific kind of buyer for this: operations managers at contact centers, at lead-generation companies, at any business where a significant portion of labor is humans doing deterministic work that could be encoded. They’ve heard about automation. They’ve seen demos. What they haven’t seen is someone who built it for themselves first, runs it in production, and can point to a floor that’s real because they certified it personally.

    The demo isn’t a slide deck. It’s Tower. It’s RALPH. It’s a system that was running while I slept and will still be running when this conversation ends.

    You can’t pitch autonomous systems credibly without having built them. You can’t build them without the warrior sessions. The sessions were never just sessions — they were the R&D for a product I hadn’t named yet.

    The Honest Accounting

    There are systems I built that aren’t running. Scripts that automated a task I stopped doing. Repos that solved a problem that no longer exists. Not every warrior session produces something that compounds — some of them produce something that was useful once and isn’t anymore.

    That’s fine. The return on the ones that do compound is high enough to cover the ones that don’t. PrivyBot is worth every session that went into it and several that went into things I’ve since discarded. ContentPipeline has scheduled more content than I’ve actively thought about. TeleseroAdmin2026 has run more dialing adjustments than I could have made manually in the same period.

    The pipeline runs while I sleep. That’s not a metaphor for anything. It’s a literal description of what happens between midnight and 6am on Tower.

    I built that. In the margins. One session at a time.

  • The Bug That Looked Like Slow, and Was Actually Broken

    It was one of those checks that should’ve taken thirty seconds. I ran a search against a real list — a few hundred leads, standard call, nothing exotic — and it just sat there. No error. No result. Just quiet.

    I assumed it was slow. I’d built the thing to hit an internal API and page through records, so “slow” was the obvious story, and I believed it for longer than I should have. I even started looking at whether I needed to add caching.

    Then I ran the same search on a smaller list — thirty records instead of three hundred — and it worked instantly. That’s when I knew it wasn’t slow. Slow doesn’t have a cliff. Broken does.

    The real problem was a single field. My data model marked customer email as an optional, validated email field — which sounds correct, and is correct, right up until the source system’s convention for “no email on file” turns out to be an empty string instead of a null. Pydantic’s email validator doesn’t know what to do with an empty string. It doesn’t skip it. It rejects it. And it rejects it silently enough, deep enough in a batch operation, that the whole search just — stopped. No traceback pointing at the actual cause. Just nothing.

    I’d been debugging the wrong problem for the better part of an hour. I was optimizing for a diagnosis I’d made before I had any real evidence for it, and once I’d said “it’s probably slow” out loud, I kept looking for reasons that were true instead of reasons that were right.

    The fix was small once I found it — one validator that runs before the email check, converting empty strings to null so the real validation logic still applies to anything that’s actually malformed. Seven new tests to make sure it stayed fixed. But the fix isn’t the lesson. The lesson is that “it’s slow” and “it’s broken” produce completely different debugging paths, and picking the wrong one costs you real time before you even notice you’re on it.

    I’ve started treating my own first explanation as a hypothesis to disprove, not a starting point to build on. The five-minute version of that discipline: before you optimize anything, prove it’s actually the bottleneck you think it is.

    Next: going back through every other endpoint in the same tool with the same question — not “is this slow,” but “have I actually confirmed that, or just assumed it.”

  • I Processed 671,000 Records in 6 Minutes and 32 Seconds

    The number that mattered wasn’t 671,000. It was 6:32.

    But before I got there, I had to survive a BOM file.

    What a BOM File Does to Your Morning

    A BOM — Byte Order Mark — is a hidden character. Three invisible bytes at the start of a UTF-8 encoded file, placed there by certain export tools as a signature. Completely benign in most contexts. Catastrophic if you’re parsing column headers programmatically and nobody told you it was there.

    The file came in as a standard monthly lead drop from a third-party vendor. CSV, normal structure, expected columns. I loaded it, ran my process, and watched it fail in a way that made no sense. The column I was looking for was right there in the header row. My code couldn’t find it.

    I opened the file in a hex editor. The first column name didn’t start with the letter I was looking at. It started with EF BB BF followed by the letter. Three bytes of invisible garbage prepended to the header, making Name into something my string comparison had never seen before and would never match.

    That was lesson one: files lie. Specifically, files produced by systems you don’t control lie in ways you won’t anticipate until they do it to you. The fix was one line. The lesson was architectural.

    Tool One: The Fuzzy Scrubber

    The lead data problem predates the BOM file. It starts with a simpler, more persistent irritant: duplicate companies.

    When you’re working with raw lead data at scale — scraped data, third-party drops, list purchases — you consistently encounter the same fundamental problem. A regional franchise has fifty locations. A corporate chain has a hundred. A national company has branch offices in every market you’re targeting. Each one appears as a separate row with a slightly different name. Smith Plumbing, Smith Plumbing LLC, Smith Plumbing of South Florida, Smith Plumbing — Boca Raton.

    You don’t want fifty versions of the same company. You want one, or none.

    The first tool I built was a fuzzy scrubber. Not exact match deduplication — exact match is easy and catches almost nothing. Fuzzy matching: similar names, above a threshold, clustered and collapsed. The goal was to identify companies that were likely regional, corporate, or franchise operations and remove them from the working set before they reached the dialer.

    The first threshold caught seven clusters. Too aggressive — legitimate distinct companies were getting collapsed. I tuned it. Three clusters. Better. Still not perfect, but better is the goal in data work. Perfect is a fiction that costs you the pipeline.

    The scrubber became step one of what would eventually be a twelve-step process. I didn’t know that yet.

    The Encoding Problem

    Every data engineer eventually learns that text encoding is not a solved problem.

    It’s solved in theory. UTF-8 is the standard. Everyone agreed. The agreement doesn’t survive contact with files produced by legacy systems, Windows-default exports, Excel users who have never thought about encoding in their lives, or third-party vendors whose ETL tools were written in 2003.

    The lead data came from multiple sources. Each source had its own encoding habits. Most of the time UTF-8 worked. Sometimes it didn’t, and the failure mode wasn’t a clean error — it was silent corruption. Characters mangled into question marks or replacement symbols. Phone numbers with invisible characters that made them unparseable. Company names with encoding artifacts that defeated fuzzy matching and left junk in the dataset.

    The encoding handler became step two. Detect the encoding before you process. Normalize to UTF-8 explicitly. Validate that the result is clean. Only then proceed.

    The BOM problem was a subcase of this. A BOM-aware reader handles it automatically. I had not been using a BOM-aware reader. I was, after the BOM incident.

    Common Columns and the First Real Pattern

    By the time I had a fuzzy scrubber and an encoding handler, I was starting to see a pattern in what the data needed across different downstream destinations.

    We run multiple dialer systems. Each dialer has its own expected column format. The CRM backend has its own schema. A lead that’s processed correctly for one system is formatted wrong for another. If you’re loading data manually into each system, this is an annoyance — you reformat before each import. If you’re trying to automate the flow, it’s a structural blocker.

    I started mapping what each system actually needed. What columns. What names. What formats. What was optional, what was required, what would cause a silent failure if missing versus an explicit error.

    The overlap was significant. Most of what dialers need from a lead record is the same: company name, contact name, phone number, address, state, some kind of category or industry tag. The differences were in naming conventions and field formats, not in the underlying information.

    This observation led directly to the most important structural decision in the whole pipeline.

    Golden Columns

    If multiple downstream systems all need roughly the same information, and the variation is in format rather than content, then there exists a canonical representation of a lead record that can be transformed into any downstream format without data loss.

    I called this the Golden Columns.

    The Golden Column set was the formal expected schema that a lead record had to conform to before it could go anywhere. Not the format any one system needed — the superset of everything any system might need, normalized to a single consistent representation. Once a record was in Golden Column format, outputting it for any dialer or the CRM was a transform, not a reconstruction.

    This was the moment the project stopped being a collection of data-cleaning scripts and started being a pipeline.

    A pipeline needs a contract. The contract defines what goes in, what comes out, and what the shape of the data is at each stage. Before the Golden Columns, I had tools. After them, I had stages. That’s a different thing. Tools are independent. Stages are composable. You can chain stages. You can add a stage without breaking the others. You can test a stage in isolation.

    The pipeline design followed from the contract almost automatically.

    Twelve Steps

    By the time I had formalized the Golden Columns, I could see the full shape of what the pipeline needed to do to take raw third-party lead data to a dialer-ready output. I wrote it out as an ordered sequence:

    Encoding detection and normalization. BOM handling. Field presence validation against the Golden Column set. Company name fuzzy deduplication. Phone number parsing and format normalization. Address standardization. State code normalization. Industry and category tagging where present. Missing field handling and defaults. Golden Column output generation. Dialer-specific format transforms. Final validation pass.

    Twelve steps. Each one a discrete, testable operation. Each one necessary. Each one the result of a specific failure or discovery from the months of one-off processing that came before.

    671,000 records. Six minutes and thirty-two seconds.

    The speed came from the architecture. When every step is a discrete operation on a structured dataset — not a row-by-row loop, not a nested conditional mess, but a vectorized operation on a typed frame — the performance is a consequence of the design, not a separate optimization pass. I profiled it anyway. The bottleneck was where I expected it: the fuzzy matching at scale. I gave it more room to work in batch rather than iterating. The number dropped.

    6:32. That became the baseline.

    The Lesson That Took Twelve Steps to Learn

    I didn’t design this pipeline. I discovered it.

    Every tool in it started as a one-off fix for a specific problem I hadn’t anticipated. The fuzzy scrubber came from the franchise duplicate problem. The encoding handler came from the corruption problem. The Golden Columns came from the multi-system formatting problem. The BOM handler came from a hex editor at 9am wondering why a column name that was clearly visible was unreadable by my parser.

    None of it was planned. All of it was necessary.

    That’s how real data infrastructure gets built in practice: not from a design document, but from an accumulation of problems that eventually reveal the shape of the system underneath them. The design document comes after, when you’ve seen enough of the problems to know what the system is actually doing.

    The danger is stopping before you write the design document. If I’d kept the twelve steps as twelve separate scripts, I’d have twelve places to maintain, twelve places to break, twelve things to run in the right order from memory. The pipeline consolidates that into one process with a contract.

    The Golden Columns are that contract. Once you have a formal expected schema for your data, you have something you can build a tool around instead of continuing to improvise around a problem.

    The twelve steps were the improvisation. The pipeline was the design.

    The Processes Aren’t Proprietary. The Problems Are Universal.

    These patterns aren’t specific to one system or one company. They’re the natural result of working with third-party lead data at scale, and the order you build them in will follow the same logic regardless of your stack or your source.

    The fuzzy deduplication problem exists everywhere franchise and regional data gets aggregated. The encoding problem exists everywhere data crosses system boundaries. The multi-system schema problem exists everywhere more than one downstream consumer needs the same upstream data in a different format.

    The specific implementation I built is tuned to a specific set of systems, specific dialer configurations, specific CRM expectations. But the architecture transfers: identify your downstream schemas, define your canonical representation, build each cleaning step as a discrete testable stage, measure the output.

    The twelve steps I landed on were the twelve problems I encountered. Your twelve steps will be different. But you’ll encounter the BOM file. You’ll encounter the franchise duplicate problem. You’ll hit the moment where two systems need the same record formatted two different ways and you realize you’ve been solving the wrong problem.

    When you get there, you need a contract. Define what a clean record looks like before you worry about what any specific system needs from it. Everything else follows from that.

  • From Pong AI to Play Store: How a Childhood Hobby Became a Rust Game Engine

    The first game I wrote with any real ambition wasn’t a game. It was a NEAT implementation that learned to play Pong. I wasn’t trying to ship anything — I was trying to understand how a system could learn to do something I taught it.

    That question has been running in the background of everything I’ve built since.

    TurboShells came next. I took the NEAT studies from PyPong and asked: what if instead of teaching an AI to hit a ball, I bred turtles? Each turtle’s body was drawn entirely from its genes — no sprites, pure math. Shell radius, leg length, color — all expressed from a genetic sequence at render time. They raced. The faster ones bred. The slower ones didn’t.

    Nobody played TurboShells. But I learned something: the genetics loop — dispatch a breeding pair, wait for the outcome, observe the consequence — was more interesting to me than any game mechanic I’d seen. I wasn’t building a racing sim. I was building a system that made things happen without me.

    ChimeraLab was the first time I tried to give the genetics a body.

    Custom 3D creature rendering. SpineComponents — oblongs stacked together, body parts articulated from code. I got a humanoid assembled. I gave it a skeleton. I ran the simulation.

    It fell to the floor under its own weight.

    I never did fix the bipedal problem. But I got it to transition from two legs to four using a slider, and watching that happen — a creature renegotiating its relationship with gravity in real time — taught me more about 3D rendering than any tutorial I’d read. Sometimes the failure is the lesson.

    rpgCore was the foundation I should have built first. A thousand tests. Real ECS architecture. Genetics, lifecycle, dispatch — everything composable, everything verified. SlimeGarden put it to work: an astronaut crash-lands on an unusual planet and finds slimes. Breed them. Dispatch them. See what comes back.

    It sounds simple. It wasn’t. And it pointed somewhere.

    Seven projects in, I was staring at the Google Play Store submission checklist.

    OperatorGame ran on Android. Real Rust, real Bevy, real APK on a real phone. The combat worked. The UI was clean. I’d solved the hard problems.

    The submission required a 512×512 app icon, a 1024×500 feature graphic, and two screenshots.

    I didn’t have any of them.

    I could have made them. It would have taken an afternoon. But sitting there looking at that checklist, I realized the assets weren’t the problem.

    The problem was I had no audience. I was about to pay the Play Store’s attention tax — discoverability weighted toward downloads, downloads toward reviews, reviews toward players who found you somehow — with zero players behind me. I wasn’t Android Store money-ready. I was Itch.io audience-ready.

    Those are different things. Confusing them is expensive.

    Here’s what the journey looked like from the inside:

    PyPong AI taught me how systems learn. TurboShells taught me that genetics loops are more interesting than game mechanics. ChimeraLab taught me that creatures fall down and that’s instructive. rpgCore gave me the foundation. SlimeGarden gave the foundation a story. OperatorGame proved the Android pipeline. VoidDrift took the dispatch loop — Scout mines ore, returns, consequence — and dressed it in something people want to watch.

    Every project is the same loop. Something dispatches. It does its work. It returns with a result. Something changes.

    I’ve been building that loop for years. I just didn’t see it until I looked at the list.

    The lesson isn’t “don’t aim for the Play Store.”

    It’s: know what you’re ready for. The Play Store is a distribution problem you solve after you have players, not before. Itch.io is where you find out if anyone cares. If they do, the Play Store is a next step. If they don’t, you learned that cheaply instead of expensively.

    VoidDrift is on Itch right now. A small audience that keeps coming back. That’s the signal I was missing with OperatorGame.

    When the audience is real, the Play Store assets take an afternoon.

    There’s a story that keeps circling my mind. Someone in a ship, traveling through a black hole, becoming something else. They find a station. What follows is a macabre exploration of self — what survived the transit, what didn’t, what the new thing is capable of.

    VoidDrift is the ship and the void. SlimeGarden is the crash-landing after.

    The loop doesn’t end at the Play Store. It ends when the story does.

    The Scout dispatch loop in VoidDrift is the same loop TurboShells was running — breed, wait, observe — dressed in space mining clothes, nine projects later. Phase 5 is live. The Play Store listing is three PNGs away.

    The story is still circling. I’m still building toward it.

  • I Built a CLI to Replace Expensive AI Directive Generation

    I Built a CLI to Replace Expensive AI Directive Generation

    The friction point is simple to describe. Claude designs the architecture. Windsurf builds it. The directive that connects them — structured, scoped, phase-gated — gets written by me, by hand, every single time.

    I’m the middleware. I built a tool to replace myself. It didn’t quite work.

    OpenAgent started from a real observation: the same context was being re-explained in every session. I had architectural patterns, stop rules, test floor conventions — and every new Windsurf session, the agent had no idea any of it existed. The directive was the missing connective tissue. Write it well and the agent stays on scope. Write it badly and the agent invents its own architecture.

    So I built a CLI that reads the codebase, understands the structure, and generates directives shaped to my development style. The breakthrough was SOUL.md — eight questions about how I actually work. That profile gets embedded in every directive. When OpenAgent generates something, it references the right conventions, names the right stop conditions. It sounds like something I’d write.

    It’s on PyPI as openagent-directive. v0.2.2. 103 passing tests.

    Here’s the part I didn’t put in the README: I still do the same manual cycle.

    The tool works. The directives it generates are useful. But I’m still the one handing them to Windsurf, watching the session, course-correcting when it goes sideways. The friction I wanted to eliminate is still there because the real blocker isn’t the directive — it’s that there’s no coding agent that lives outside an IDE.

    Windsurf, Cursor, Copilot — they all require a human in the seat. The autonomous loop I wanted, where OpenAgent feeds a directive to a coding agent that executes independently, reports back, and waits for the next one, doesn’t exist yet in any reliable form. The IDE-bound constraint kills the automation before it starts.

    I built a correct solution to the wrong layer of the problem.

    The pivot I keep thinking about: OpenAgent as an MCP tool. Not a CLI that generates directives for humans to hand off, but a codebase intelligence layer that a coding agent can query directly. What files are in scope? What’s the test floor? What patterns does this codebase use? An agent with access to that context doesn’t need a human to write the directive — it can construct its own.

    That version of OpenAgent is waiting on the ecosystem. When a capable coding agent exists that can operate outside an IDE, receive a structured task, execute against a real codebase, and return proof — OpenAgent is already positioned to be the interpreter it needs.

    For now it’s a CLI on PyPI that saves me twenty minutes per directive and reminds me that some problems can’t be fully solved until the infrastructure around them catches up.

    The friction is still there. The tool is ready when it isn’t.

  • What Ants Taught Me About Systems Design

    What Ants Taught Me About Systems Design

    Before programming, there were ants.

    Not metaphorical ones. Real colonies — pheromone trails that appear from nothing, queen mortality absorbed without panic, chambers organized by workers who’ve never been given instructions. No central controller. No plan. Just rules simple enough to follow and complex enough to produce something alive.

    When I started building software, I was chasing the same thing.

    AntSim is an evolutionary ant colony simulation with split-view rendering. The top two-thirds is the foraging ground — pheromone trails building and fading in real time. The bottom third is the colony cross-section: chambers, tunnels, brood, storage.

    The genetics system controls five traits per ant: sensitivity, speed, boldness, lifespan, energy efficiency. That’s it. Five genes. The colony doesn’t know it’s evolving. Individual ants don’t know they’re part of a system. They follow their traits and the colony either survives or it doesn’t.

    When a queen dies — randomly, the way queens do — workers autonomously identify royal jelly candidates and raise a successor. Colony continuity without external control. The system just continues.

    The original genetics system had too many genes. Forty variables, every trait interacting with every other. The ants evolved into creatures that couldn’t survive their own complexity — biologically sophisticated and functionally useless.

    Stripping back to five traits was uncomfortable the way all simplification is uncomfortable. It felt like giving up. What I got instead was emergence — behaviors I didn’t design, produced by interactions I didn’t anticipate, between five simple numbers and enough time.

    The cross-section visualization is still in progress. Phase 3 rendering complexity exploded the moment I made it detailed. Individual chambers, tunnel geometry, precise brood locations — the simulation could handle it but the renderer couldn’t. I scaled back. It’s still on the list.

    Here’s what I didn’t understand when I started: I wasn’t building a simulation. I was building a philosophy.

    Every system I’ve built since has the same underlying shape. Something dispatches — an ant, a Scout drone, a queued job, a breeding pair. It does its work without supervision. It returns with a result. The colony, the station, the codebase, the task queue — they change accordingly.

    That loop appeared first in an ant colony. It’s in VoidDrift’s Scout dispatch. It’s in PrivyBot’s job queue. It’s in rpgCore’s genetics engine. I’ve been building the same system for years in different materials.

    Emergent behavior doesn’t require complex rules. It requires simple constraints, honest consequences, and enough time to surprise you.

    The ants figured that out long before I did.

  • The Engine Legacy: From Asteroids to rpgCore

    The Engine Legacy: From Asteroids to rpgCore

    At some point I stopped counting how many times I’d written the same movement system.

    It wasn’t any one project’s fault. Every game had its own physics, its own collision logic, its own state management — each a reasonable decision in isolation, collectively a pattern I was tired of repeating. The tenth time you write an entity update loop you start asking a different question: not “how do I build this game” but “what would make the next one easier to start.”

    rpgCore started as a refactoring project and became something else entirely.

    The idea was simple: extract the patterns that kept repeating and build a reusable toolkit. A simplified Entity-Component-System in Python — modular enough that adding a genetics system didn’t require touching the movement code, adding a lifecycle system didn’t require touching the genetics code. Each piece independent, each piece composable.

    Simple ideas scale badly. The toolkit grew. A thousand tests. Then eleven hundred. GeneticsComponent. LifecycleComponent. DispatchSystem. ResourceFlow. A scene system, a UI theme, a demo registry. At some point it stopped being a toolkit and started being a codebase worth protecting.

    The mistake I kept almost making was treating rpgCore as a means to an end. It isn’t. The foundation is the thing.

    When you build something generic enough to support any idea, you start seeing which ideas are worth having. The genetics system in rpgCore is the same system that ran TurboShells’ turtle breeding. The dispatch pattern is the same one VoidDrift’s Scout drones use. The resource flow is the same loop that appears in SlimeGarden. rpgCore didn’t just save me from rewriting code — it made the pattern visible, and once the pattern is visible you can apply it deliberately.

    The 1,122 tests aren’t there because I’m disciplined. They’re there because a foundation without verification isn’t a foundation, it’s a guess.

    The BreedingSystem is still queued. Allele resolution, trait inheritance, phenotype expression from genotype. When it ships it connects to every genetics-adjacent idea I’ve had since TurboShells. ConquestSystem is queued behind it.

    rpgCore doesn’t ship. It makes shipping possible.

    That’s a slower kind of value than a finished game. It’s also the only kind that compounds.

← rfditservices.com