out/stability-B2.md
Saved at B2.B2d: gold and stability check. This reading copy substitutes participant identifiers in saved text; it is not a new interview response. Reading Markdown.
Participant handles in these reading copies use the pixel stream's identifiers: base64(SHA1(username)), case preserved, without a salt. These are linkable pseudonyms. Profile links are removed; missing-author markers remain. Original research files are unchanged. Quote offsets refer to original text. Source and reading-copy hashes are listed separately.
Source commit: 986c0bf4891c, 2026-10-12T16:09:56+02:00. Saved-state date: 2026-10-12T14:09:45Z. Catalogue close receipt Commit date and saved-state date are distinct.
Original GitHub file. Original SHA-256: e8c8b7fb890cbea2002f1bef90046eb35dea895d725670daace60ba8ccd1d257. Reading-copy SHA-256: e8c8b7fb890cbea2002f1bef90046eb35dea895d725670daace60ba8ccd1d257.
Stability, stage B2
Written at stage B2, leg B2d, 2026-10-12. The stability check (FLEET.md) serves the gold set to the same reader arm under two rewordings of the brief that change no test. The arm's disagreement with itself is the floor every later figure is read against.
What was served
- Set. The 40 items of
gold_items.jsonl, on the frozen scheme (scheme_final.jsonl). - Readers. deepseek/deepseek-v4.1-flash, called directly, medium effort, max_tokens 128,000, the readers of rounds 2 and 3.
- Packets. Three packets of 14, 14 and 12 items, three fresh readers each. All three rounds (gold, stab-1, stab-2) used seed 20261014. The same items fall in the same packets in the same order, so only the brief differs.
- Briefs.
- The gold round used the builder's own brief.
- Stab-1 used
out/brief-stab-1.md, a rewording of that brief. - Stab-2 used
out/brief-stab-2.md, a second rewording.
- Content of the briefs. No brief names a theme, a test or a placement.
- Returns. 9 valid returns each, none exhausted, 120
lines a round. All lines were translated back to room ids with 0 id
problems (
out/_analysis/b2_translate_fleet.py).
The stability figure
out/_analysis/b2_score_final.py computed it. For each
item, every stab-1 reader's set is compared with every stab-2 reader's
set by Jaccard, nine pairs, and the nine are averaged. The mean over the
40 items is the stability figure.
Stability: 0.807 (primary placements alike in 0.853 of the cross pairs).
The next table sets it beside each round's agreement within itself.
| figure | value |
|---|---|
| stab-1 against stab-2, mean Jaccard over the nine reader pairs per item | 0.807 |
| the same, primary placement alike | 0.853 |
| pairwise agreement inside stab-1 | 0.800 |
| pairwise agreement inside stab-2 | 0.756 |
| pairwise agreement inside the gold round | 0.818 |
| gold round against stab-1 | 0.826 |
| gold round against stab-2 | 0.796 |
| alpha over the six stability readers | 0.784 |
| alpha over all nine readers | 0.794 |
| mean change of an item's fidelity between stab-1 and stab-2 | 0.076 |
What it shows. Readers given differently worded briefs agree with each other (0.807) as much as readers given the same brief do (0.800 and 0.756). The rewording adds no disagreement beyond what three fresh readers already show under one brief. So about one placement in five differs between readers whatever the wording. That share is the floor: a later figure within it says nothing about the scheme.
Fidelity under the rewordings. It barely moves: 0.624 for gold, 0.636 for stab-1, 0.631 for stab-2. The primary match with my placement is 0.800 in all three.
Items under 0.5 across the rewordings. Five items: cuboulder t04, mariokart t04, mit t01, overwatch t23 and rit t01. Each is an item the gold readers already split on, or a two-act item where readers add or drop one theme. None moves because of a word in the brief.
What this figure licenses
This figure bounds whether the instrument decides, not whether it is right. It says how much of the readers' disagreement belongs to the readers rather than to the scheme or the brief. It is agreement with my own placements and with each other. In FLEET.md's terms, it supports the claim that the fleet produces what I would have produced, at a scale one analyst cannot reach. It does not support the claim that any competent reader would have produced it.