Skip to content
Dark Cosmos — Dev Docs

Running & testing

Running & testing (moved from CLAUDE.md)

Section titled “Running & testing (moved from CLAUDE.md)”

Moved verbatim from CLAUDE.md on 2026-09-27. Read before running Godot, exporting, or using any test/QA harness.

Running & testing

Section titled “Running & testing”
  • Merge gate: bash scripts/gate.sh runs the import pass, the full suite + test-count guard (both determinism baselines included), the tutorial trace (lessons: ok) and a 600-frame boot smoke; exit 0 = mergeable. It does NOT cover ui-sweep, soak or a device playtest. Run it in a worktree before merging, and on main after.
  • Restarting a gate: kill it by PID, never pkill -f "<worktree>/scripts/gate.sh". The gate is launched as the relative scripts/gate.sh, so a path pattern matches nothing and the old run keeps going; two gates then share one worktree (and $TMPDIR output), running against files you are still editing, and the FAIL you read is a mix of both (2026-10-16, rc-1.1-combined). Find them with pgrep -f scripts/gate.sh + lsof -p <pid> | grep cwd, kill children (pkill -P <pid>) then the PID.
  • TestFlight release: scripts/release/release.sh (no argument): the store build number IS Sim_Const.BUILD. It refuses to run if a store-<BUILD+1> tag exists, bumps BUILD + all four presets via scripts/bump-build.sh, commits the bump before building (a failed run burns that number; the next run takes N+1), builds + uploads iOS and tvOS, waits for processing, then tags the bump commit store-N and pushes branch + tag atomically. Default is internal only: the Internal group (hasAccessToAllBuilds) gets every build with no review. EXTERNAL=1, or asc.py beta N later, attaches it to the external groups and submits beta review; do that only once the build is checked on a device, since external testers get every build sent (NO_UPLOAD=1 stops after the .ipa). Pieces: build-ios.sh, build-tvos.sh (the four tvOS re-apply fixes), asc.py builds|wait N|beta N|feedback (asc.py next-build is diagnostic only: max ASC build + 1, NOT the next store build). Needs the sandbox disabled. asc.py feedback --out DIR pulls every TestFlight note (all pages) into a numbered checklist; work from that, not a hand-copied list.
  • Project subagents (.claude/agents/, model pinned so they don’t fall back to the Haiku default): bh-implementer (sonnet since 2026-10-16 trial, see CLAUDE.md § subagent models; writes code, runs the gate, never pushes), bh-reviewer (opus, adversarial pre-merge review), bh-visual-qa (sonnet, device-size captures), bh-feedback-triage (sonnet, all TestFlight + D1 feedback into one checklist).
  • Godot 4.7.1: CLI godot (/opt/homebrew/bin/godot), editor /Applications/Godot.app (upgraded from 4.6.3 via brew upgrade --cask godot 2026-07-19 — GUT vendored addon bumped 9.6.0→9.7.1 alongside it, macOS/Web export templates installed to match, see docs/architecture/godot-gotchas.md). This is the stock editor only — the separate ~/Claude/godot-tvos custom fork (Metal-patched, “4.7.dev”) used for the actual App Store archive is untouched by this upgrade.
  • A real macOS deploy target exists (export_presets.cfg preset “macOS”, universal binary, codesign/codesign=0): godot --headless --path . --export-release "macOS" builds/macos/BulletHeaven.app then open builds/macos/BulletHeaven.app (or run the binary directly at .../BulletHeaven.app/Contents/MacOS/Bullet Heaven). builds/ is gitignored. No signing/notarization needed for local-only running — the export template ships an ad-hoc embedded signature and a freshly-exported .app carries no com.apple.quarantine xattr (that flag only gets added by browsers/Mail/AirDrop, not local export tools), so Gatekeeper doesn’t block it. Distinct from “open in the editor and press F5” below — this is an actual exported, standalone build, matching what ships to the TV/iPhone, useful for verifying a release build’s behavior (not editor-debug behavior) on the Mac itself. ⚠️ export_presets.cfg (root-level, holds this “macOS” preset + “Web”) is gitignored and does NOT exist in a fresh worktree checkout — copy it from the main checkout (cp /Users/chris/Claude/bullet-heaven/export_presets.cfg .) before exporting from inside a worktree.
  • Play it: open in the editor and press F5, or godot --path . (or just godot from the project dir).
  • Agents/scripts launch every WINDOWED Godot through scripts/lib/godot-bg.sh <godot-binary> [args] (or godot_bg after source scripts/lib/godot-bg.sh), never bare. Godot on macOS activates itself (activateIgnoringOtherApps, the unbundled-activation hack, makeKeyAndOrderFront) and steals the keyboard focus from whatever Chris is typing in. The wrapper injects scripts/lib/godot-bg-shim.m (built on demand to ~/Library/Caches/bh-godot-bg/, cached by source hash; no engine change) which no-ops those calls, forces the Accessory activation policy, orders the window front without making it key, parks it bottom-right (BH_BG_POSITION=br|bl|tr|tl|none), and holds an App Nap activity so frames keep advancing. --headless runs, BH_FOREGROUND=1, and a failed shim build pass straight through. ui-sweep.sh, ui-probe.sh, ui-capture-store.sh, pipeline-tour.sh and godot_export already use it, and godot-window-lock.sh puts a PATH shim first so a bare godot inside a locked command is backgrounded too. Shader compile check (CLAUDE.md § headless gate): bash scripts/lib/godot-bg.sh godot --path . --rendering-method mobile --quit-after 300 and the same with --rendering-method gl_compatibility, under scripts/dev/godot-window-lock.sh when other agents run too, then git checkout -- project.godot audio/bus_layout.tres. Prove a launch is focus-safe with scripts/dev/verify-bg-focus.sh <binary> <args> (samples the frontmost app every 100 ms; --wrapped <cmd> for a script that wraps itself). scripts/test/godot-bg-guard.test.mjs fails the gate when a script/tool launches windowed godot outside the wrapper. DYLD injection works because /Applications/Godot.app and the fork editor both allow it (do not strip their allow-dyld-environment-variables entitlement); SIP strips DYLD_* across /usr/bin/env/bash/perl, so it is only ever set as an argument to env right before the godot exec.
  • ⚠️ Any WINDOWED godot --path . run (even a throwaway one-off .tscn for visual verification) can silently corrupt project.godot — reformats/mangles hand-authored comments into unreadable run-on garbage, drops sections ([physics] vanished once), reverts config/features’s engine-version tag. The ROOT project.godot has stayed clean across headless runs, BUT platform/tvos/project.godot was rewritten by a plain headless --import on 2026-09-30 (reordered keys, boilerplate header prepended, config/features tag flipped) — check git status after ANY Godot invocation, root or tvOS, windowed or headless. Always git diff project.godot after any windowed invocation and git checkout -- project.godot immediately if it shows changed — never let it land in a commit. Full story: docs/architecture/godot-gotchas.md.
  • Fresh worktree checkouts need one import pass first: godot --headless --path . --import — otherwise GUT errors with “Some GUT class_names have not been imported” on the very first test run. One-time per worktree.
  • Full test suite (GUT 9.6.0, headless): godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gdir=res://tests -ginclude_subdirs -gexit (exit 0 = pass).
  • Single test: -gtest=res://tests/<file>.gd does NOT narrow the run — .gutconfig.json hardcodes dirs: ["res://tests"], which silently wins over -gtest/-gdir (verified: still runs the whole suite, not a narrowed subset). Use -gselect=<script_name_without_.gd> instead (e.g. -gselect=test_areas — confirmed narrows to that one script). ⚠️ -gselect is a SUBSTRING match over script names: -gselect=test_tunables also matches test_tunables_registry (2 scripts ran, 2026-09-30). Use the exact name including .gd (-gselect=test_tunables.gd) when two scripts share a prefix.
  • Test-count guard: scripts/check-test-count.sh runs the suite and fails loud if GUT ran fewer scripts than there are tests/test_*.gd files (the stale-class-cache silent-drop). Self-maintaining (no hardcoded count); use it in CI / before relying on a green run. Captures the run’s output with || true and judges purely by parsing GUT’s own summary (Scripts count + a derived failure count), so a nonzero godot exit code can’t abort it silently before the count check runs — fixed 2026-07-04 (NEXT_TASKS.md), re-verified working end-to-end 2026-07-19 on the 4.7.1 upgrade below.
  • Godot’s headless run used to SIGABRT (exit 134) at engine teardown even on a fully green suite — a known upstream GDScript shutdown crash, godotengine/godot#119279, reproducible on both 4.6.3 and 4.7-beta. Fixed upstream (maintainer-merged PR, closed 2026-07-13) and confirmed gone as of the 4.7.1 upgrade below: headless runs now exit 0 cleanly. Full story: docs/architecture/godot-gotchas.md’s 2026-07-19 entry, NEXT_TASKS.md.
  • Headless boot smoke: godot --headless --path . --quit-after <frames> — boots + ticks the real sim; grep stderr for SCRIPT ERROR.
  • Internet co-op relay, end to end: scripts/relay-local-test.sh [ticks] [idle_s] (defaults 3600, 65; RELAY_PORT env, default 8787). Runs the relay Worker under npx wrangler dev --local (workerd, the real Durable Object runtime; never deploys) and headless Godot clients against it over a real WebSocket, using the online dev CLI flags (--mp-host-online=CODE, --mp-join-code=CODE, --mp-relay=ws://127.0.0.1:PORT, --mp-auto, --mp-ticks=N, plus --mp-build=N to make a joiner claim another BUILD). Five checks, each printed PASS/FAIL: an idle host lobby stays connected for idle_s; a wrong-BUILD joiner ends rejected: build mismatch… (the host’s reason survives the relay’s 4005 kick); the next joiner right after that kick is relay peer 2; host + joiner play ticks ticks with equal final tick/checksum, no desync, RTT measured on both and a level-up; a killed host ends the joiner’s session host_left within 1.5 s. The parity check between FakeRelayHub (GUT) and the real runtime; NOT part of scripts/gate.sh (needs npx, a free port, ~3 min). Run it before any device test of a relay/ or relay net/ change, and before merge. It restores project.godot / audio/bus_layout.tres itself. The LAN/online CLI itself is documented in docs/architecture/coop-and-pilots.md.
  • QA harnesses (built 2026-07-14, RITUAL GATES during the v0.1 RC):
    • scripts/soak.sh --runs=N --ticks=T --ships=a,b --coop=both [--invuln=1] [--player=skilled] [--seed0=S] — headless autopilot soak: invariants (NaN/HP-bounds/softlock/early-end) + per-weapon damage attribution (Sim.stats_sink + ProjPool.context stamps) + cross-run zero-damage anomaly detection; JSON report under gitignored reports/. Any sim-behaviour change → one soak run must come back clean. --player=skilled (added 2026-07-16, see Current status) swaps in SkilledAutoPlayer and adds a fun_signals section to the report (death-tick distribution, build diversity, upgrade dud/overpowered rates, the pre-boss/boss anticlimax ratio) — a numeric proxy for “is this fun,” not just an invariant check; default player is still the original dumb AutoPlayer, unchanged. --invuln=1 (dev-invuln pilots) is the coverage mode that reaches bosses/dimensions/late-game; mortal fresh-save runs die at 1–3 min. Softlock flags on solo runs ~7 min in are usually a >3-min solo boss duel (arena empties in PHASE_BOSS), not a real stall — check before chasing.
    • scripts/ui-sweep.sh [--bless] — screenshot regression vs 9 blessed baselines (tools/ui_sweep/baseline/, RMSE 0.06). Run WINDOWED + FOREGROUND only (headless can’t render MultiMesh colours; backgrounded windowed godot gets App-Napped). --bless copies the EXISTING out/ — eyeball every PNG there first. Any UI-affecting change → sweep must pass. ⚠️ It is #!/bin/zsh — invoke it as zsh scripts/ui-sweep.sh. Under bash it dies on setopt/print and reports “sweep did not complete”, which reads like a capture failure. ⚠️ As of 2026-08-19 this gate FAILS on unmodified main — bestiary.png (stale baseline, needs re-blessing) and hud_in_run.png (a live in-run frame that differs from ITSELF by ~0.09 across two runs of identical code, i.e. no threshold can make it meaningful). Before treating a red sweep as your regression, re-run it once to gauge that capture’s own variance, then A/B against a detached worktree at main (git worktree add --detach <tmp> main; copy the gitignored export_presets.cfg in). Tracked in NEXT_TASKS.md.
    • Sim cost A/B: godot --headless --path . -s res://tools/sim_bench/sim_bench.gd -- [--loadout=a,b,c] [--phases] — seed-1234 invulnerable Survival, 3600 ticks x 3 reps, prints late-half µs/tick plus the state checksum and the final active= weapon ids. Run it on two commits (copy the file into a git worktree add --detach <commit> checkout, --import there first); the checks must match or the two runs did different sim work. Without --loadout it flies the starter Blaster only, so weapon update cost is barely visible. --loadout=nova,orbit,scar_tissue,momentum_core,turret,scatter makes that list the EXACT active set in update order (the Blaster is dropped unless listed), granted before tick 0 through UpgradeSystem.grant_weapon, so at most max_weapon_slots (6) ids; the bench exits 1 if any id was not granted. A loadout run is a separate baseline with its own checksum: compare loadout runs only with loadout runs. Per-weapon cost without --phases: run one weapon at a time (--loadout=turret) and diff µs/tick against --loadout=blaster. --phases adds a per-phase µs/tick table (every once-per-tick module call, W:<wid> per weapon, plus an “unwrapped” remainder). It is timed from outside /sim: the bench builds runtime subclasses of Sim and each module/weapon script and swaps them in with set_script, so nothing in /sim carries a timer or flag. Only the outermost wrapped call is charged. The timers add overhead, so take µs from a plain run and the split from --phases, and check its checksum equals the plain run’s. A phase renamed in /sim makes --phases exit 1 (update the lists at the top of the bench). tests/test_sim_bench_harness.gd pins both options. Run-to-run noise on an M5 is around ±8% of µs/tick, so take the median of several interleaved runs.
    • godot --headless --path . -s res://tools/story_auto/story_trace.gd -- [--mode=tutorial|story] — the ONLY automated coverage story/tutorial mode has (soak.sh drives Survival only). Prints every dialogue beat with its tick, gap, line count and running kills/level-ups, and exits 1 if the run never finished. Use it after ANY change to data/story.json, sim/story_director.gd or the tutorial gates. It runs on StoryAutoPlayer (tools/story_auto/), which exists because SkilledAutoPlayer structurally cannot complete a story run — with no enemies alive its move vector is ~zero so it never leaves the entry corridor, and it flees a boss indefinitely rather than killing it. Both failures look like game bugs in a trace. ⚠️ Gaps in the output are SIM time and exclude the tutorial’s read-freeze, so they are not wall-clock pacing — cadence and feel still need a human.
    • Input replay: Debug Settings → “Dev: record replays” (dev build / editor, takes effect next launch; windowed only, headless runs ignore dev flags), play, and the session saves to user://replays/ as a lossless fixture (ReplayLoader re-ticks it to an identical checksum). Since 2026-10-14 the file also carries the level-up picks and dimension_picks (older files reproduce only up to their first level-up). Headless playback: ReplayDriver (input/).
    • Human replay vs autopilot: godot --headless --path . -s res://tools/replay_compare/replay_compare.gd -- --replay=<file> [--telemetry=<runs row json>] [--bot-seeds=10] rebuilds a fresh-save human replay (picks from the file, else from the telemetry row’s timeline_json.levelups, each offer checked), reports fidelity against the truth, and logs per-second movement features for the human and SkilledAutoPlayer (attract + raw planner). Findings: docs/research/2026-10-14-toby-replay-vs-autopilot.md.
  • New code is TDD’d against GUT; render/feel is verified by playing (or a runtime MCP — see below).