B2 · Coding: pilot
Test whether others can use the scheme
A category can seem clear to its author and confuse everyone else. B2 tests that problem on a small set before the scheme is used across the whole collection.
Where this stage starts
B1 has supplied provisional categories and rules for assigning material to them. B2 starts with those rules, known confusing pairs, and both difficult and randomly selected examples. “Placement” means assigning a local theme to one or more of the common categories.
What the aggregator does
The author records their own answers before seeing anyone else’s. Fresh readers then receive the scheme and examples, without the author’s working notes or answer key. Three readers examine each pilot item. Their replies show where the rules lead to different interpretations.
The aggregator asks why each important disagreement arose. Was a rule vague? Did one local theme bundle several kinds of action? Did a reader miss evidence? Was the author’s original answer wrong? These explanations matter more than making an agreement score rise. The scheme can be revised, and every revision must have a recorded reason.
The pilot also tests whether small changes in the wording of instructions alter placements and whether readers behave differently near the beginning and end of a batch. This helps determine how much material later readers should receive together.
Rounds stop under the declared rule: no further edit, the declared agreement threshold, or the four-round limit. Any remaining weakness stays in the report. A separate set of reference answers is then used to check the frozen scheme. These are the author’s judgments, sometimes called “gold” answers; they are not an external truth about the communities. Agreement measures consistent use of the instrument, not whether its interpretation is correct.
What the next stage receives
B3 receives one fixed version of the scheme, a reply format and evidence about batch design. Freezing the rules allows disagreements in the full reading to be compared under the same instructions. It does not make those rules immune to later criticism.
What the output files contain
Open a file group for its purpose. These are descriptions of the stage’s outputs. Markdown files hold prose; JSONL files hold one structured record per line.
The reference answersauthor_items.jsonl, gold_items.jsonl
author_items.jsonl records the author’s pilot placements
before comparison. gold_items.jsonl holds the separate
reference items and judgments used to assess the frozen scheme. Their
role is a declared comparison standard.
The reader instructionsRETURN-B2.md
RETURN-B2.md defines what a reader must return. It makes
replies comparable and lets the handler distinguish an incomplete reply
from a substantive disagreement.
What each round changedpilot_rounds.jsonl, scheme_edits.jsonl, exemplar_candidates.jsonl
pilot_rounds.jsonl records the rounds and results.
scheme_edits.jsonl links each rule change to its reason.
exemplar_candidates.jsonl identifies difficult examples
that might help readers in the full reading.
Stability and the overall assessmentstability-B2.md, pilot_report.md, notes-B2.md
stability-B2.md reports sensitivity to different
wording. pilot_report.md brings together agreement,
reference matching, remaining confusions and practical limits.
notes-B2.md records settings, thresholds and decisions.
The frozen scheme and its provenancescheme_final.md, scheme_final.jsonl, run_manifest.json
scheme_final.md is the readable scheme;
scheme_final.jsonl is the equivalent structured record used
by tools. The stage’s run_manifest.json entry records how
the pilot was run, so later results can be tied to the tested setup.
Full stage description
Read the complete technical card for exact tasks, reading boundaries, checks and closing rules. The explanation above is a guide; the card holds the instructions.
How the stage is divided into legs
B2a: build round. Repeats: round. Interview point: -.B2b: check round. Repeats: round. Interview point: -.B2c: gold and stability build. Repeats: no. Interview point: -.B2d: gold and stability check. Repeats: no. Interview point: -.
Source versions and hashes identify the exact technical documents reproduced here.