Reliability and reproducibility paths

Scoring Reliability

What independent scorers receive

• The same frozen run artifacts, including task description, constraints, trace, validation output, and final deliverables.
• A fixed scoring rubric for Plan Quality, Assignment Quality, Coordination, Deliverable Quality, Efficiency, invalid-run flags, and failure labels.
• Calibration examples that show how evidence maps to score bands without revealing the official ranking.
• An anonymized scoring CSV template using framework aliases such as F01, F02, and F03.
• A conflict rule confirming the reviewer did not design EPOB, generate the run evidence, or write the benchmark claims being assessed.

Scoring Output

What the reliability study produces

• Raw independent reviewer CSVs are preserved before averaging or adjudication.
• Inter-rater reliability is reported for continuous category scores where enough paired ratings exist.
• Invalid-run and failure-label agreement are reported separately from numeric score agreement.
• Adjudication notes explain large disagreements by returning to the trace and artifact packet rather than changing the rubric after the fact.
• The paper may claim independent scoring reliability only for the reviewed sample and only after these statistics are complete.

Experimental Reproducibility

What an independent replication needs

• The same task instances, dataset slice, scorer version, timeout policy, and evidence schema.
• The same model, seed, harness, container, and resource profile used by the frozen comparison cell.
• Access to the benchmark runner, container image or Dockerfile, environment template, and non-secret configuration manifest.
• A requirement to preserve fresh run traces, final artifacts, hashes, logs, and aggregate comparison outputs.
• A disclosure path for blocked cells, invalid runs, provider drift, model-endpoint changes, or hardware differences.

Workflow

One-page onboarding checklist