Fidelity harness: per-build report card

Fidelity harness: per-build report card

One command, run once per BUILD, gives the whole headless evaluation suite a single verdict: run everything (balance/feel/difficulty probes, soak, sim_bench, story_trace), diff every metric against the previous build, flag anything outside its band, and post one summary to the tracker. Built for issue #115 on device_sim’s scorecard model; design: docs/superpowers/specs/2026-10-22-per-build-report-card-design.md.

A green card is not “the game is good” — it means the measured numbers did not move past their bands. A WARN is a flag for a human, never a build block on its own.

Run it

scripts/dev/report-card.sh                      # full suite (run per BUILD; takes a while)
scripts/dev/report-card.sh --suite quick        # smoke suite: proves the harness in minutes
scripts/dev/report-card.sh --list               # the plan: items + argv; runs nothing
scripts/dev/report-card.sh --only soak,balance  # iterate on one or two items
scripts/dev/report-card.sh --post               # also post the card to the tracker hub issue
scripts/dev/report-card.sh --init-hub           # once: create the hub issue, write tracker.json

Exit 0: no fail row (warns are flags). Exit 1: any fail. Exit 2: harness error (bad suite/bands, --import timeout, interrupted — nothing is posted).

What one run does

1. Reads the BUILD number from sim/constants.gd and the git rev (like ui-capture-store.sh, before any import pass can dirty the tree).
2. Runs one godot --headless --path . --import pass (fresh worktrees hang on the stale class cache otherwise), with gate.sh’s snapshot-restore of project.godot + audio/bus_layout.tres.
3. Runs each suite item (tools/report_card/suite.d/full.json) in its own dir under reports/report-cards/build-<N>/<timestamp>/<key>/: the tool’s argv (fixed seeds/ticks, so every build measures the same thing), stdout+stderr to run.log, results parsed to result.json (report-item/1).
4. Diffs against the newest stored card from a previous BUILD with the same suite_id (--base-dir to read elsewhere), applies tools/report_card/bands.json, and renders card.txt + writes card.json (report-card/1).

Reading the card

== REPORT CARD build=336 suite=full-1 git=1e3b60f7 duration=812s
-- SUITE (7 items)        per-tool status + one-line headline
-- DIFF vs build 335      metric, current, previous, delta (+pct), band verdict (ok/warn/fail/better)
-- FLAGS                  every warn/fail from the bands, with the numbers
-- ITEMS NOT CLEAN        items whose own verdicts failed (feel band FAIL, soak invariants...)
-- CAVEATS                what each tool cannot tell you
== VERDICT: WARN (...) exit=0

Tool-native verdicts count: a feel band FAIL makes the feel item WARN (a finding, not a failed run), soak invariant failures make soak FAIL, a failed story trace fails its item. hard_tuning_probe always exits 0 and sim_bench/story_trace only print to stdout, so the harness trusts their parsed output/JSON over exit codes (a crash exit ≥ 128 or a timeout fails the item regardless).

Where results live, and the baseline rule

• Cards: <repo>/reports/report-cards/build-<N>/<YYYYMMDD-HHMMSS>/ (reports/ is gitignored; the tracker comment is the permanent record). Run from the main checkout to accumulate history; a worktree run starts its own (point --base-dir at the main checkout’s dir to cross-read).
• The diff baseline is strictly a lower BUILD with the same suite_id: re-running the same build diffs against the previous build’s card, not against itself. Changing suite params means bumping suite_id, which deliberately resets the diff until a new baseline exists.

Posting to the tracker

--init-hub creates one log issue (“Per-build report cards (BUILD summaries)”) via scripts/dev/file-issue.sh and saves its number in tools/report_card/tracker.json (committed, so every clone posts to the same hub). --post adds one comment per build+suite with the full card in a code fence and a hidden <!-- report-card:build-N:suite-S --> marker; a re-run edits its comment in place instead of stacking duplicates. Quick-suite smoke cards are clearly labelled by their suite_id (quick-1).

Gotchas

• The feel bands are calibrated for the full caps (ticks=18000, 300 s); the quick suite’s feel numbers are smoke only — that is why quick.json has no story_story either (a truncated story trace can never finish, a designed fail).
• soak and the difficulty probe write fixed unix-named files under reports/; the harness snapshots that dir before/after and captures the one new file. Two concurrent probe runs in the same checkout make it refuse (capture_error) rather than misattribute.
• balance/feel run with --config=live by default: numbers quote the checked-in remote-config/ record this BUILD would serve, not the live Worker KV (per the probe contract).
• The card does not gate merges (scripts/gate.sh does) and runs no GUT tests; it measures the game, not the code.