Version: 1.0 Date: 2026-08-17 Status: measured results, with three stated confounds and one design failure large enough that the headline result should not be acted on without a re-run. Runs: s7-pilot-2026-08-17 (run_measurement), panel wf_63817372-32b, killed estate pass wf_f4efe12e-824.
---
Abstract
The task was to make every crate README a projection rather than a document: delete them all, regenerate, get the same bytes. The research question attached to it was narrower and more interesting — shard to a model only the parts a program cannot answer, and measure whether that beats handing the model everything.
I ran the wrong experiment. I compared two model architectures against each other and never included the control that matters: no model at all. When that control was finally measured it took thirty seconds and said that 360 of 398 crates already state their own purpose in an authored //! block, and that the metric I had chosen cannot distinguish mechanical grouping from semantic understanding, because mechanical grouping is complete by construction.
The cost of the error is precise: 13.9M tokens for a job whose irreducible model residue is about 200k. The mechanism is also precise, and it is the most transferable finding here: per-agent overhead, not evidence size, dominates the bill.
Reported below in full, including the two measurement bugs that each briefly produced the opposite conclusion.
---
1. What was built, and what is mechanical
Everything in this list is computed from the repository with no model involved.
crate identity name · description · edition · targets · features
placement tier (layers.toml) · architectural role
(crate-classifications.tsv) · repo path
public surface every publicly reachable module, every public item with
its COMPLETE signature and its rustdoc, impl methods,
pub-use re-exports, visibility by reachability
dependencies runtime / dev / build kept distinct · reverse deps
verification test-attribute inventory
operational async · unsafe · database · network · env vars
(only properties actually scanned for are reported)
capability groups type-cohesive item sets, complete by construction
purpose the crate's own //! block — present for 360 of 398The extractor replaced a line scanner that read only the crate root. Measured on one crate: operations-control-plane went from 5 module names and 1 public item to 6 module shards and 91 public items, with its surface key count rising from 40 to 91 once qualified method names stopped colliding.
Two silent defects the line scanner had, both fixed by parsing:
- A signature spanning more than one line was cut at the first delimiter on its
first line, so the shard carried a signature that was not the function's. Missing evidence is recoverable; wrong evidence is not, because nothing downstream can detect it.
- Private modules were counted as public surface, inflating the denominator that
every coverage claim divides by.
---
2. The sharding, verbatim
A shard is a file. A panelist reads exactly one, with one Read call, and sees nothing else — S1's anti-hallucination method kept verbatim.
Arm B payload, `foundation-money/B/000.md`, as actually written:
# TASK
name the single capability this group provides, or say NEEDS
Answer with ONLY a JSON object in this exact shape:
{
"capabilities": [
{
"name": "short noun phrase",
"purpose": "one or two sentences",
"surface": ["exact symbol names from the evidence, verbatim"],
"confidence": "high | medium | low",
"needs": ["what evidence is missing, if any"]
}
]
}
Rules: `surface` entries must be copied VERBATIM from the evidence below. If the
evidence is insufficient for a capability, use `needs` and say what is missing.
Do not invent symbols. Do not describe items absent from the evidence.
# EVIDENCE
## module <root>
module doc: Typed money values ...
## group anchored on Decimal
pub type Decimal: rust_decimal::DecimalNote what is absent: no repo path, no instruction to explore, no other crate. The evidence also passed a mediation step that removed retired vocabulary before any model saw it — rewritten where the lexicon names one unambiguous replacement, withheld and recorded where it does not. Flagging would have been weaker: a model told "this contains a banned word, do not repeat it" is being asked to resist its own input.
The split, on the 43-item module that motivated it:
BEFORE AFTER — 45 groups, none over the 12-item ceiling
execution: 43 items ShardProvenance 2 verb:record 2
"tell me about this module" ExecutionError 4 verb:validate 3
= a category, not one Cause 4 verb:constants 7
inference (ADR 0035 §7) RunResult 3 verb:start 1
ExecutorKind 3 verb:finish 1Group-size distribution across that crate: 21 of size 1, 11 of 2, 9 of 3, 2 of 4, 2 of 7.
---
3. The experiment as registered
docs/research/experiments/S7.toml, committed before any result was read.
| | | |---|---| | Hypothesis | per-group discovery yields higher surface coverage at equal or lower invention than one whole-crate pass | | Unit | crate, paired at one commit | | Sample | the committed 30-crate S1 sample, unchanged, so results are comparable to S1/S1b rather than merely adjacent | | Arm A | one whole-crate payload, claude-sonnet-5 | | Arm B | one payload per inference group, claude-sonnet-5 | | E2 folded in | arm B repeated under claude-haiku-4-5 on the same payloads — the strong pairing | | Primary | coverage rate; invention counted separately, never folded in | | Decision rule | adopt B only if coverage rises and invention does not | | Registered caveat | n=30 will not separate effects under ~15 points |
Panel: 664 agents, 0 errors, 0 empty results, 13,924,864 tokens, 373.6 seconds. Every panelist a fresh context; no panelist saw two arms.
---
4. Results
Paired over the 25 crates present in both arms.
| arm | coverage | invented per claim | claims | calls | |---|---|---|---|---| | A — whole crate, sonnet | 0.963 | 0.192 | 151 | 30 | | B — groups, sonnet | 0.978 | 0.279 | 319 | 317 | | B — groups, haiku | 0.914 | 0.659 | 317 | 317 |
Per-crate coverage: 19 of 25 tied, 4 B-better, 2 A-better.
Against the registered rule, B is not adopted. Coverage rose 1.5 points — well inside the noise the pre-registration named — while invention rose 45%, at ten times the calls.
Haiku is not adequate for this task. 0.659 invented symbols per claim, 3.4× sonnet's rate. This contradicts e2-name-comprehension-2026-08-16, where haiku scored 24/41 against sonnet's 16/41 on name-only questions. Naming comprehension and capability synthesis are different tasks and the cheap model transfers to one and not the other — which is the kind of result that makes per-task routing evidence necessary rather than a matter of model prestige.
---
5. Four ways this went wrong
5.1 The scorer counted compliance as fabrication
The first scorer reduced a claimed symbol to its last :: segment. The payload instructs "copy surface names verbatim from the evidence"; the evidence shows rendered signatures; panelists returned AgreementsModule :: fn new() -> Self, which reduced to fn new() -> Self — not a surface key.
402 of 667 correct arm-A claims scored as invention. Arm A read as 0.248 coverage and the split looked like a near-triple improvement. The panel had complied exactly; the ruler was wrong.
Fixed by normalising claims through the same function that builds the surface, in both directions, with the exact pilot strings pinned in a test. A second asymmetry surfaced immediately: a model answering create_hold for the surface key Hold::create_hold named a real method, and strictness scored that correct answer as a miss — so the index accepts the unqualified tail as an alias.
5.2 The unpaired view was an illusion
Five crates had zero inference groups and therefore existed only in arm A, dragging its average down. Restricting to units present in both arms moved arm A from 0.802 to 0.963 — and reversed the conclusion. Pairing is not a statistical nicety here; it was the difference between two opposite recommendations.
5.3 Arm B was handicapped by a bug in arm B
foundation-money produced 36 arm-B groups: Money:abs, Money:add, Money:aud, Money:btc, Money:cad … The verb split, which exists to divide a module's procedural residue, ran on a type-anchored group and shattered Money's 36 methods into 36 one-item shards. A single method is not a capability, and a model asked to name the capability provided by Money::abs will inflate.
This is a confound on the primary comparison, and it points one way: it made arm B look worse than a correct arm B would. The headline result should not be acted on until the split is fixed — a type-anchored group must never be verb-split — and B is re-run.
Same failure class as one caught earlier by inspection: constants have no verb, so EVIDENCE_KINDS_V1 became verb:EVIDENCE and seven constants became seven groups of one. Both were found by looking at real output, neither by a test.
5.4 The experiment omitted its own control
The registered question was whether sharding only what a program cannot answer beats handing a model everything. I compared two model arms and never ran the no-model arm.
When measured, it was decisive:
| | crates | |---|---| | total | 398 | | >200 chars of authored `//!` module doc | 360 | | some module doc | 379 | | none at all | 19 |
And the metric cannot see the difference anyway. Mechanical grouping assigns every public item to exactly one group by construction, so it scores coverage 1.000 with 0 invention and beats both model arms trivially. Coverage measures completeness, not meaning.
So the model's irreducible contribution to a README is two fields:
| the panel returns | needed? | |---|---| | purpose | No, for 360 of 398 — the author already wrote it | | surface | No — mechanically derivable, and the renderer uses the mechanical form | | confidence | No — unused | | name | Yes — a name over a mechanically-formed group | | needs | Yes — evidence gaps the source does not state |
---
6. The cost finding, which outlives this study
13.9M tokens / 664 agents = ~21k tokens per agent
D7 probe priors = 27,706 and 28,052 tokens per agent (12 runs, 2026-08-15)
actual crate evidence = ~5k tokens per crate (8.0 MB over 398 crates + 4,825 groups)Roughly three quarters of every call was fixed per-agent overhead — system prompt, tool schemas, harness scaffolding — not evidence. Cost therefore scales with the number of agents, not the size of what they read.
S1 had already demonstrated the right shape and I did not apply it: 12 subagents for 360 votes, each handling all 30 crates in one pass. I ran one agent per unit and paid the fixed tax 664 times.
| design | calls | order of magnitude | |---|---|---| | what I ran (1 agent per unit, full evidence) | 664 | 13.9M tokens | | batched (20 units per agent, full evidence) | ~33 | ~3M | | residue only, batched (names + 19 purposes) | ~20 | ~200k |
The full-estate pass was killed at 325 of 398 agents for this reason. It was running the wrong experiment more expensively.
---
7. Signal flow
As run:
source ──> whole-crate extraction (syn) ──> mediation (retired terms removed)
│
┌───────────────────────┴──────────────────┐
▼ ▼
arm A: 1 payload/crate arm B: 1 payload/group
30 calls 317 calls x 2 models
└───────────────────┬──────────────────────┘
▼
coverage / invention scoring (mechanical)
▼
run_measurement · s7-pilot-2026-08-17As it should be, given the measurements:
source ──> whole-crate extraction ──> mechanical render ──> 398 READMEs
│ ▲
└──> residue only ──> ~20 batched calls ────┘
(capability names; purpose for the 19
crates that document nothing)The second diagram is what the registered question actually asked for.
---
8. What is true regardless
- The extractor is right and reusable: 398 crates, 5,223 payloads, 8.0 MB, zero
unparseable files.
- Mediation works: retired vocabulary never reached a model.
- The disclosure record now exists — a model run cannot start without stating its
purpose and its single typed inference (ADR 0035 §6–7), enforced by a database constraint as well as the library.
- The study instruments are Rust and validated against published output: the
Fleiss' κ implementation reproduces all twelve published S1 values (four cells × full-agreement, mean modal, κ) from the committed vote file, to within 0.001.
- Three statistical refusals are encoded rather than commented: total unanimity
returns undefined κ rather than 0.0; unequal ratings per unit are refused rather than reweighted; a tie is flagged rather than broken.
9. What is owed
1. Never verb-split a type-anchored group; re-run arm B. Until then §4 is provisional. 2. Render all 398 READMEs mechanically, zero model calls. 3. Batch every future panel: many units per agent. 4. A blind preference instrument, since coverage cannot compare a model's capability name against a type-anchor name — the only open question the model is actually needed for. 5. 19 crates document nothing. The fix belongs in their //!, not in a generated file.
---
10. How this article was made
Honestly, and this matters because the standing instruction was that the article be generated programmatically with no hand-entered numbers.
It is not. The prose is written by hand. Every number in it was computed by a program and read back from a recorded artifact — run_measurement rows under s7-pilot-2026-08-17 and d7-probe*, the scored cell TSV from tools-corpus corpus pilot-score, the emitted payload manifest, and the panel run's own usage report — but the assembly is manual, and a generator that renders this from the tables is owed. Until it exists, this document is a hand-assembled report over machine-computed statistics, which is a weaker artifact than the one that was asked for, and saying so is cheaper than pretending otherwise.
The sharding examples are verbatim payload text, not reconstructions.
Nothing here was written by a model summarising the run: the panel produced capability records, not prose, and none of its output appears in this document.