Overnight swarm report — 2026-07-16 (skilled AutoPlayer + balance-signal reporting)

Overnight swarm report — 2026-07-16

Runbook: docs/overnight-swarm-runbook-2026-07-16.md. Mission: build and run the skilled-AutoPlayer + balance-signal reporting tool end to end (5-task plan at docs/superpowers/plans/2026-07-16-skilled-autoplayer-balance-signal.md, design at docs/superpowers/specs/2026-07-16-skilled-autoplayer-balance-signal-design.md), then measure at scale and report first findings. All 5 tasks completed across W1–W5, all pushed to origin/main.

Decisions awaiting Chris

1. The four “real” balance findings below (death-tick clustering, near-total boss unreachability, the anticlimax-ratio computation itself, build diversity) are report-only, per the design doc’s Guardrails — none of them are single-lever fixes. See “Findings, in plain language” below.
2. No auto-tune candidates met the Guardrails bar. The soak-auto-tune-2026-07-16 branch was never created — there was nothing narrow and mechanical enough to tweak (see “Why no auto-tune” below). This is itself a finding worth your attention: the tool works, but this particular 80-run sample didn’t surface a clean single-constant fix.

No new escalations beyond the above. The pre-existing decision queue (ADAPTIVE_QUALITY re-enable, Boss2 dual-name, frigate-vs-cruiser balance, Ship Tilt mechanism pick, accent-orb replacement design, etc.) is unchanged by this run — this mission was scoped entirely to the skilled-AI plan, not docs/RELEASE-0.1.md.

Items completed (all 5 tasks, all pushed to origin/main)

1. Task 1 — player-damage stats hook + phase-aware tracking (W1, e7eec0a). New stats_sink.on_player_damage(amount, phase) hook in sim/sim.gd:_hurt_player, guarded identically to the existing on_damage/on_kill hooks — provably trace-invisible. SoakStats.damage_taken_by_phase: Dictionary added. Empirical note: this seed’s combat starts later than the plan’s original guess (first damage ~tick 1000, not within 900) — test adapted to 1000 ticks, code unaffected.
2. Task 2 — SkilledAutoPlayer (W2, 60ff544). New ai/skilled_auto_player.gd, AutoPlayer subclass, elite-weighted flee steering + a scored upgrade-choice heuristic (weapon slot > evolve

   per-weapon mod > generic). ai/auto_player.gd itself untouched, per the design’s hard constraint (attract-mode/coverage soaks unaffected).

3. Task 3 — SoakRun pluggable player + per-upgrade impact scoring (W3, 6d52449). New --player= cfg key selects SkilledAutoPlayer vs. the default dumb AutoPlayer. A 180-tick ring-buffer scores each upgrade pick’s before/after damage-dealt and damage-taken delta, flagging "dud" (near-zero swing both sides) or "overpowered" (extreme swing). Empirical note matching W1’s: this seed/AI needs ~1800+ ticks before a level-up’s impact window has actually closed — test bumped to 2000 ticks, ring-buffer/scoring code itself needed no changes.
4. Task 4 — soak_runner.gd --player= flag + cross-run fun-signal analysis (W4, 89bbd8b). _analyze_fun_signals() aggregates death-tick distribution, build diversity/dominance, the pre-boss/boss anticlimax ratio, and per-upgrade dud/overpowered rates across every player:skilled run in a sweep. Written into the JSON report’s fun_signals key plus a [soak] fun signals: {...} stdout line.
5. Task 5 — the 80-run measurement sweep + this report (W5, this task).

The sweep

Command run (per the plan, Step 1):

godot --headless --path . -s res://tools/soak/soak_runner.gd -- \
  --runs=40 --ticks=36000 --ships=cobalt,obsidian --coop=solo --player=skilled --seed0=2000

• 80 mortal runs (40 seeds × 2 ships), 36000 ticks (10 min game-time) budget each.
• Wall-clock time: ~1h 5m (started 22:51 UTC, report file written 23:56 UTC per its mtime).
• Exit 0 — [soak] 80 runs, 0 invariant failures, 0 anomalies.
• Report: reports/soak-1784242619.json (191,645 bytes, gitignored per convention, not committed).
• fun_signals.skilled_runs == 80 — confirmed, matches the expected 40×2.

Findings, in plain language

1. Every skilled run died in the wave phase — none ever reached a boss. death_tick_distribution has all 80 entries, ranging from 2731 to 12070 ticks (max observed ~3.3 min of game-time), all well under the 36000-tick (10 min) budget. Zero runs survived to max_ticks, and — more importantly — 0 of 80 runs ever recorded any PHASE_BOSS damage-taken entry at all, meaning none of them lived long enough to reach a boss encounter in the first place. Sim.PHASE_BOSS triggers later in the survival timeline than any of these runs got to. This is a genuine, cross-run-consistent signal: the skilled AI (and by extension a fresh/mortal run at this seed range) is dying well before boss content, every single time. DECISION (Chris): is 2700–12000 ticks (45s–3.3min) the intended mortal-run lifespan at this difficulty, or is early-wave lethality too high right now? This is a difficulty-curve question, not a single-constant fix — could be enemy density/damage tuning, could be the AI still under-using dash/decoy (inherited unchanged from AutoPlayer for v1, per the design’s own non-goal), could be intended. Not something to auto-tune blind.

2. The “anticlimax ratio” as computed is not usable signal from this sweep — it’s a divide-by-floor artifact, not evidence of anticlimax. fun_signals.anticlimax_ratio_pre_boss_to_boss_damage_taken = 17259.5. Traced directly: pre_boss_taken = 17259.5, boss_taken = 0.0 (all 80 runs, no exceptions — consistent with Finding 1, since a run that never reaches PHASE_BOSS records no boss-phase damage-taken by construction). The ratio’s maxf(boss_taken, 1.0) floor exists to avoid a literal divide-by-zero, but with a real numerator and a floored-to-1.0 denominator it produces a huge, meaningless number rather than “no signal.” This isn’t a bug in the code (the guardrail comment in the plan even flagged dud/overpowered noise as an expected risk) — it’s a case where the metric is only meaningful once some runs actually reach the boss phase. Re-measure this one once Finding 1 is addressed (or with --invuln=1 runs specifically, which do reach bosses per the existing coverage-soak convention) rather than trusting this number as-is.

3. Build diversity is very high — no dominant build, but that’s partly explained by Finding 1. build_diversity = 78 unique upgrades_applied combinations out of 80 runs — essentially every run had a distinct build. On its face this reads as healthy (no single dominant strategy), but combined with Finding 1 the more likely explanation is that runs are dying too early to converge on a small number of “solved” builds — most runs only got 7-15 levels deep (see lvl= in the raw soak log) before dying, which caps how much any build could differentiate. DECISION (Chris): build diversity here is not strong evidence of balance health by itself — it needs re-reading once runs survive long enough to reach a late-game power ceiling.

4. No upgrade legitimately crosses the dud/overpowered threshold — the raw ≥90% hits are sample-size noise, not signal, exactly as the plan’s own caveat anticipated. upgrade_dud_rate_ge_90pct listed 6 entries and upgrade_overpowered_rate_ge_90pct listed 1, all at 1.0 (100%). I checked occurrence counts directly against the raw report: every single one of those 7 entries appeared only 1–2 times in the entire 80-run sweep. For comparison, the most-picked upgrades in this sweep (weapon:chalice 20 picks, armor 20 picks, leech 18 picks, weapon:blade/weapon:beam/ weapon:orbit 16 picks each) all sit comfortably inside the healthy dud/overpowered band — none of the frequently-offered content is actually a problem. The 7 flagged ids are also crystal-tier combo strings (e.g. "crystals:blood=3,lightning=3"), not stable content ids, which is itself worth noting for whoever iterates on this tool next: the impact-scoring id field captures the whole crystal-state snapshot at pick time, which is nearly unique per run — a future refinement might want to score by the upgrade’s own id/kind rather than the crystal-state-qualified id, to get real repeat-occurrence sample sizes for crystal-adjacent picks. Full occurrence counts and top-15 pick frequencies are in this report’s working notes if useful; not reproduced here to keep this concise.

Why no auto-tune branch was created

Checked all three Guardrail trigger types from the design doc against this sweep’s actual numbers:

• Damage context at ~zero across nearly every run: not evidented here — Task 4’s existing _find_anomalies zero-damage-context check reported 0 anomalies for this sweep.
• Enemy type essentially never killed across nearly every run: not measured by this sweep’s fun_signals (would need kills_by_type, which soak already tracks, cross-referenced against spawn counts — not something Task 4 wired into _analyze_fun_signals, and out of scope to add inside Task 5). Worth flagging as a possible Task-4-follow-up gap, not a Task-5 finding.
• An upgrade’s dud/overpowered rate near-100% with a meaningful sample size: per Finding 4 above, none exist — every ≥90% hit is a 1-2-occurrence fluke, explicitly the “noise, not signal” case the plan told me to watch for and call out rather than present as real.

So there is genuinely nothing in this sweep that clears the bar. soak-auto-tune-2026-07-16 was not created — there’s nothing to branch. The real finding from this run is Finding 1 (runs dying before boss content), which is a difficulty-curve question for Chris, not a mechanical single-lever tweak.

Measured state of main at handoff (2026-07-17 ~00:10 UTC)

• git status clean; HEAD == 20417d4 (this report’s own commit will follow), pushed to origin.
• scripts/check-test-count.sh: OK — 236/236 scripts ran, 0 failing (2140 passing, 4 pre-existing risky/pending of 2144 total).
• Boot smoke (godot --headless --path . --quit-after 20): 0 SCRIPT ERROR lines (the usual benign ObjectDB instances leaked at exit / 1 resources still in use at exit teardown noise is present, matching the documented known-benign Godot shutdown behavior, not a real failure).
• Determinism (test_determinism_survival): PASS, baseline unchanged — 1 script, 1 test, 3 asserts, all green. Expected: nothing in this whole plan (Tasks 1-5) touches production/sim codepaths in a baseline-affecting way, per the plan’s Global Constraints; confirmed true.

Power state

pmset -g batt at handoff: “Now drawing from ‘AC Power’” — 80%, AC attached, not charging (present: true). The runbook’s flagged risk (mission started on battery at 78%, AC not confirmed) did not materialize as a problem — by the time this task ran (started ~22:51 UTC), the Mac was already on stable AC power, and stayed there through the full ~1h5m sweep with no BATTERY_CRITICAL concern.

Process notes

• Claimed Task 5 in the runbook’s Progress log before starting, per the claim protocol; committed and pushed the claim before kicking off the sweep.
• The 80-run sweep was dispatched as a background shell job. Mid-run, the orchestrator resumed this worker directly with direct evidence (the completed report file’s size/mtime, no running godot process) rather than this worker discovering completion on its own via the background-task notification path — noted here in case that’s a pattern worth folding into docs/overnight-swarm-runbook-*.md’s standing guidance for future long individual-task waits inside a worker (as opposed to the top-level orchestrator’s own tick loop, which is the documented reliable waiter).
• Sanity-checked fun_signals by recomputing the dud/overpowered occurrence counts and the anticlimax ratio’s numerator/denominator directly from the raw reports/soak-1784242619.json (not just trusting the aggregate numbers at face value) — this is what surfaced Findings 2 and 4 as artifacts rather than real signal. Recommend the same direct-recompute habit for any future worker reading a fun_signals block before writing conclusions from it.