A plain-language account of twenty-four hours in which a software team treated vocabulary as an engineering material, built a machine to package work for AI, and watched the AI's honesty find bugs in the machine. Written 2026-08-15; every number here comes from a logged measurement, not memory.
---
The problem nobody writes down
Every software team has a private language. Ours had a word — call it "the A-word" — that we used constantly, and it turned out to mean two completely different things: sometimes a kind of business object (a customer, a contract), sometimes a sequence of processing steps (fetch, extract, index). Nobody noticed, because humans resolve that kind of ambiguity without thinking.
Our newest team members can't. They're AI models, and for a reader that is a model, a fuzzy word isn't a style problem — it's a defect that spreads. The model guesses, the guess lands in a document, the next model reads the document, and the guess hardens into "fact." We had already watched this happen: a suffix invented for one special file got copied onto thirty-three files in nine days, meaning nothing on thirty-two of them.
So we stopped arguing about words and started measuring them.
The experiment: put the dictionary on trial
The setup is simple to describe. Take thirty real components of our codebase. Write each one an index card of plain facts. Hand the cards to a panel of AI judges — each one fresh, isolated, seeing nothing but the cards — and ask each judge to classify every component using vocabulary A (our homemade terms) or vocabulary B (the industry's established terms). Then measure how often independent judges agree.
That agreement score is the whole trick. A good word is one that different readers resolve the same way; a bad word scatters them. You don't debate it — you count it.
Twenty-one judges and 540 votes later:
- Our homemade vocabulary and the industry one tied — until we noticed the
industry set had no word for "developer tool," which our set did. We added one word, re-ran the panel, and the amended industry vocabulary won outright, posting the best agreement score of the whole study.
- The ambiguous A-word split exactly as predicted: the judges unanimously sent one
of its meanings to one industry term and the other meaning to another. One word had been two all along.
- One behavior worried us: judges were allowed to say "I don't know" and never
did — not once in 540 votes. Remember that; it comes back.
By afternoon the winning vocabulary was ratified and wired into the repository's own commit checks, so the retired words now physically cannot re-enter our documents without a recorded exception. That's the house philosophy in one line: a rule is a wish; a mechanism is a fact. We'd already proven it the hard way — in an earlier session, the one document arguing for a word ban itself contained the banned word, and only the automated check caught it.
The mess we then had to clean
All this machinery guards the gate to our code repository, and the gate was reporting 88 problems: stale statistics pasted into documents by hand, one dead link, five pairs of planning documents accidentally sharing the same number, and about forty code components with no README at all.
The old way to fix this is to point an AI at the repository and say "clean it up" — and hope. We did the opposite, and this is the method half of the story.
The method: index cards, not the library
We wrote a small program called a work-package compiler. Its one job: read the gate's list of 88 problems and sort every problem into one of three bins.
1. A program can fix this. Stale numbers, renames, dead links — a machine knows the right answer exactly. No AI involved. (This turned out to be most of it.) 2. A tiny choice is needed. A dead link with several plausible targets — an AI may pick one, from a list, and nothing else. 3. Actual writing is needed. The missing READMEs. Only here does an AI write prose — and only from a shard.
A shard is the index card from the experiment, made rigorous. For each code component, a gathering program mechanically extracts: its name, its manifest description, its own documentation lines, the signatures of its public functions (never their internals), its optional features, its module names, which other components use it, what layer it lives in, its version, and whether it has tests. Three hundred ninety-three cards, built by a program in seconds.
Just as important is what a shard never contains: the component's source code bodies, anything about other components, and anything about our conversations. The AI writer gets the card and a contract: four sections, twenty-five lines, and if the card doesn't contain a fact you need, write "NEEDS: that fact" — never invent it. The card is the writer's entire world. It cannot leak what it was never given, and it cannot hallucinate a claim we can't trace, because every claim must point back to a line on the card.
The plot twist
Here's the part we didn't expect. When we ran the cleanup for real, the AI writers were never called. A dumber tool — one that assembles READMEs directly from each component's own existing description — covered all forty components. The "judgment" bin, the one bin we'd reserved for intelligence, emptied itself. Machines fixed the numbers, machines renamed the files and rewrote every reference, machines assembled the docs. 88 problems became zero, and the final upload to GitHub passed every automated gate running live, no overrides.
Along the way the gates caught four bugs — two in the very tools we'd just built — which is exactly the point. Nobody was careful. The machinery was.
So we tested the AI path anyway
An untested path is a promise, not a result. We pulled six representative cards and gave them to six isolated AI writers, then graded their READMEs against the machine-assembled ones.
Round one: zero invented facts across all six — every sentence traced back to the card — and six honest "NEEDS" notes where the card genuinely lacked a fact (the card format had no version or test-status field). One writer over-did it, flagging "NEEDS" on a field that was present but empty. Both problems were card problems, not writer problems. So we fixed the card: added the two missing fields, and added one sentence to the contract — an empty field is a fact; report it as "none."
Round two, fresh cards: zero invented facts again, zero NEEDS, six complete READMEs. And then the best moment of the whole day: two writers truthfully reported "version: empty," which was our bug — the card-maker didn't understand components that inherit their version from the workspace. The AI's honesty found a defect in the instrument measuring it. We fixed the card-maker the same hour.
Remember the panel judges who would never say "I don't know"? That's why this matters. You cannot ask a model to be uncertain and expect it to stick — our own decay study shows written rules bend wherever they cost something. But you can build the uncertainty into the package: a closed contract, a card that is the whole world, and a NEEDS escape valve that is cheaper to use than inventing. Honesty held for twelve straight generations not because the models were virtuous but because nothing pulled against it.
The numbers, all in one place
- Words: 540 classification votes, 21 judges; best agreement score 0.914 after the
one-word amendment; classification confidence rose across three vocabulary generations: 0.912 → 0.945 → 0.967.
- Cleanup: 88 gate findings → 0; five colliding documents renamed under the
operator's "add another decimal" rule with every cross-reference rewritten; final push passed all gates live.
- Shards: 393 cards gathered mechanically; 12 AI-written READMEs across two
probes; 0 invented facts in all 12; NEEDS notes 6 → 0 after the card fix.
- Bugs found by machinery during the day: 6, including 3 in the day's own new
tools. Disk reclaimed from stale test debris: 142 GB.
What it adds up to
The day's quiet thesis: with an AI on the team, language and process are the same engineering problem. Words get measured and ratified like interfaces. Work gets compiled into packages the way source gets compiled into binaries — with the smart part confined to small, sealed, checkable slots. The industrial precedent (Palantir's platform runs on a similar creed: the AI proposes, the machinery executes) says this scales; our contribution is doing it in the open, inside an ordinary git repository, with the receipts committed next to the code.
The next time someone on the team reaches for a word, there's a five-line checklist waiting. The next time work needs an AI, there's a card. And the next time anyone claims the system works — there's a gate that will want proof.