Skip to content
Dark Cosmos — Dev Docs

Per-build report card design

Per-build report card, design

Section titled “Per-build report card, design”

Issue #115 (owner-agent). Direction OK’d by Chris 2026-10-22: build on tools/device_sim’s scorecard model, diff each build against the previous one, post one summary to the tracker. Source idea: docs/research/2026-10-22-agent-evaluation-tooling-ideas.md idea 2.

Why

Section titled “Why”

The repo has many headless evaluation tools (balance/feel/difficulty probes, soak, sim_bench, story_trace) but no single verdict per build. Every number lands in a different place (probe JSON, reports/, stdout) and nobody diffs them against the previous build, so regressions surface only if a person happens to run the same probe twice and remember the old value.

What it is

Section titled “What it is”

One command, run once per BUILD (at a build bump or release prep, from the repo root):

scripts/dev/report-card.sh                    # full suite, writes card + prints it
scripts/dev/report-card.sh --post             # ... and posts the summary to the tracker hub issue
scripts/dev/report-card.sh --suite quick      # smoke suite (harness validation, small seeds)
scripts/dev/report-card.sh --only balance,soak
scripts/dev/report-card.sh --list             # plan only
  • Suite config: tools/report_card/suite.d/full.json + quick.json — one entry per tool with its exact argv (fixed seeds/ticks so every build measures the same thing), timeout and parser. Changing an entry’s params changes suite_id, which deliberately breaks diff comparability until a new baseline exists.
  • Runs each item in its own dir under <out>/<key>/ (run.log + parsed result.json), process-group kill on timeout (no timeout(1) on macOS), one --import pass first (fresh-worktree stale class cache) with the gate’s project.godot/bus_layout.tres snapshot-restore.
  • Parsers normalize every tool to {metrics: {name: number}, facts: {name: string}} (report-card/1). Balance/feel read their probe-result/1 JSON (authoritative status), soak and difficulty read their captured reports/ JSON (both write fixed unix-named paths, so the harness snapshots reports/ before/after and captures the one new file), sim_bench/story_trace parse stdout + exit code.
  • Diff: against the newest stored card from a previous BUILD with the same suite_id, metric by metric. Bands (tools/report_card/bands.json): relative diff bands per metric ({"worse": "lower", "warn": 0.10, "fail": 0.25}) plus absolute rules (invariant_failures >= 1 → fail, lessons_ok == 0 → fail). Tool-native verdicts surface too: feel band FAILs, soak anomalies. Anything warn/fail lands in a FLAGS section.
  • Storage: <repo>/reports/report-cards/build-<N>/<YYYYMMDD-HHMMSS>/card.json + card.txt (reports/ is gitignored; local history, tracker comment is the permanent record).
  • Posting: --post comments on one dedicated hub issue (created by scripts/dev/report-card.sh --init-hub via scripts/dev/file-issue.sh), one comment per build+suite, idempotent via a <!-- report-card:build-N:suite-S --> marker (edit in place on re-run). The card’s posted field records issue + comment URL.
  • Verdict + exit: FAIL/1 if any item or band failed, WARN/0 if only warnings, harness error → 2. A warn does not block a build; it is a flag for a human.

Out of scope (v1)

Section titled “Out of scope (v1)”
  • No CI/Actions trigger — the card runs on the local machine (GitHub Actions minutes are a shared pool). The merge/deploy ritual just tells you to run it.
  • No absolute calibrated bands for balance metrics yet (diff-based until the numbers are calibrated); bands.json is the file to grow.
  • No bot fidelity (#114) or replay upload — separate issues; this card is where their numbers will report once they exist.