Skip to content
Dark Cosmos — Dev Docs

Persona benchmark: evolved elites vs the default genome (2026-10-13)

Persona benchmark: evolved elites vs the default genome

Section titled “Persona benchmark: evolved elites vs the default genome”

Question: do the learning autopilot’s evolved elites survive longer than the plain default genome (today’s fixed ProfiledAutoPlayer AI) at the same skill?

Short answer: at strong skill, yes, clearly for 3 of 4 cells (tank, support, utility; +59 to +163 s per run on paired seeds). At median skill only the tank elite is better (+24 s), and at weak skill no elite is distinguishable from the default. The other 8 cells all point the elite’s way except damage:strong (−3.4 s), but each is within noise at 20 seeds. No cell is measurably worse than the default.

Method

Section titled “Method”
  • Tool: tools/persona_evolve/benchmark/benchmark.mjs (+ PersonaBench, bench_runner.gd), see docs/architecture/learning-autopilot.md § Benchmark.
  • Archive: production GET /personas, fetched 2026-10-13 14:20, epoch 2, 12 elites (10 found by the soak learner on BUILD 311, tank:median and tank:strong by an Apple TV on 312/313).
  • Each elite and the default genome (PersonaGenome.new()) at the cell’s skill flew the same 20 fixed seeds (PersonaBench.SEED_BASE = 20261013; seeds 1085003918..37), under exactly the learner’s conditions (EvolveBatch.soak_persona: fresh MetaState, default ship, survival, no co-op, humanize + throttle, max_ticks 72000 = 20 min). No run reached the cap.
  • The default genome does not depend on the target role, so it was flown once per skill and paired with all four of that skill’s cells. 299 runs, 8 parallel godot processes, 230 s wall (1614 s CPU) on the M5.
  • Stats: paired difference elite − default per seed, mean with a 95% CI from the paired t (n−1 df: 19, or 18 for damage:strong), two-sided p, wins-losses (the rest are exact ties). “Role played” = PersonaJobs.role_of for the elite’s own cell’s role.

Results (survival seconds)

Section titled “Results (survival seconds)”
Cell Elite (archive score) n Elite mean ± sd (s) Default mean ± sd (s) Paired diff (s) 95% CI p W-L Role played Verdict
tank:weak Tank-Weak #4 (64.9) 20 57.6 ± 40.9 44.0 ± 21.4 13.6 -0.1 … 27.2 0.051 12-5 tank 11, utility 9 within noise
tank:median Tank-Median #2 (75.3) 20 64.2 ± 46.0 40.6 ± 18.4 23.6 4.9 … 42.3 0.016 15-3 tank 14, utility 6 elite better
tank:strong Tank-Strong #2 (178.5) 20 171.0 ± 88.7 112.4 ± 76.8 58.6 15.5 … 101.6 0.010 15-4 tank 19, utility 1 elite better
support:weak Support-Weak #4 (64.5) 20 63.0 ± 52.7 44.0 ± 21.4 19.0 -4.6 … 42.7 0.108 11-9 damage 9, support 7, utility 4 within noise
support:median Support-Median #2 (78.9) 20 52.7 ± 33.4 40.6 ± 18.4 12.1 -7.3 … 31.5 0.209 12-6 damage 7, support 7, utility 3, tank 3 within noise
support:strong Support-Strong #2 (155.4) 20 275.8 ± 224.1 112.4 ± 76.8 163.4 53.6 … 273.2 0.006 18-1 support 9, damage 8, tank 3 elite better
damage:weak Damage-Weak #3 (52.0) 20 46.1 ± 27.8 44.0 ± 21.4 2.1 -13.6 … 17.9 0.779 8-10 damage 8, support 5, utility 4, tank 3 within noise
damage:median Damage-Median #3 (58.5) 20 48.2 ± 29.4 40.6 ± 18.4 7.5 -0.4 … 15.5 0.061 5-2 damage 15, utility 4, tank 1 within noise
damage:strong Damage-Strong #3 (129.8) 19 112.2 ± 68.2 115.6 ± 77.5 -3.4 -58.7 … 51.9 0.898 10-9 damage 12, support 4, utility 3 within noise
utility:weak Utility-Weak #4 (54.6) 20 50.1 ± 30.9 44.0 ± 21.4 6.2 -3.7 … 16.0 0.206 12-7 tank 10, utility 10 within noise
utility:median Utility-Median #3 (65.1) 20 48.2 ± 29.4 40.6 ± 18.4 7.5 -0.4 … 15.5 0.061 5-2 damage 15, utility 4, tank 1 within noise
utility:strong Utility-Strong #4 (172.6) 20 186.8 ± 115.2 112.4 ± 76.8 74.4 9.7 … 139.0 0.026 14-6 utility 14, tank 6 elite better

Per skill, the best elite (highest benchmark mean among that skill’s 4 cells) vs the default. Picking the best of 4 after the fact flatters the elite, so read the per-cell table for significance:

Skill Default mean ± sd (s) Best elite (cell) Best elite mean ± sd (s) Paired diff (s) 95% CI W-L Cells with CI > 0 / < 0
weak 44.0 ± 21.4 support:weak 63.0 ± 52.7 19.0 -4.6 … 42.7 11-9 0 / 0 of 4
median 40.6 ± 18.4 tank:median 64.2 ± 46.0 23.6 4.9 … 42.3 15-3 1 / 0 of 4
strong 112.4 ± 76.8 support:strong 275.8 ± 224.1 163.4 53.6 … 273.2 18-1 3 / 0 of 4

Archive mean (what the Worker recorded) vs this benchmark’s elite mean, same genome:

Cell Archive mean (samples) Benchmark mean (20 seeds)
tank:weak 80.4 (10) 57.6
tank:median 107.5 (3) 64.2
tank:strong 225.7 (6) 171.0
support:weak 91.1 (4) 63.0
support:median 100.1 (5) 52.7
support:strong 192.5 (8) 275.8
damage:weak 84.0 (4) 46.1
damage:median 64.7 (23) 48.2
damage:strong 162.1 (12) 112.2
utility:weak 67.3 (9) 50.1
utility:median 72.1 (6) 48.2
utility:strong 208.6 (12) 186.8

Verdict, plainly

Section titled “Verdict, plainly”
  1. Strong skill: the learner is working. tank:strong (+59 s, CI 16..102), support:strong (+163 s, CI 54..273, 18 wins of 19 decided seeds) and utility:strong (+74 s, CI 10..139) beat the default on the same seeds. damage:strong is a coin flip (-3 s, CI -59..+52): that elite has not learned anything measurable.
  2. Median skill: one real win. tank:median +24 s (CI 5..42). The rest are within noise. The damage:median and utility:median cells hold the same genome (2e2fbd1853014dec, identical rows) and it ties the default exactly on 13 of 20 seeds, so in most runs it plays like the default.
  3. Weak skill: no detectable gain. All four point positive (+2 to +19 s), tank:weak just misses (CI -0.1..27, p 0.051), but none is distinguishable from the default at 20 seeds.
  4. Multiple comparisons. 12 cells were tested. The 4 “elite better” verdicts survive a 10% false- discovery-rate correction (Benjamini-Hochberg) but none survives a strict family-wise Holm/Bonferroni at 5% (smallest p = 0.006 vs 0.0042 needed). Treat the strong-skill wins as solid in direction and size, not as proven individually. 11 of 12 cells have a positive mean difference, but the cells share their skill’s default runs, so that count is not 11 independent tests.
  5. Archive scores are optimistic. 11 of 12 elites scored lower here than in the archive (often by 30-50%): winner’s curse from promoting on 3-12 lucky samples, which is exactly what score = mean − SE is meant to damp. Not all of the gap need be luck: the elites were scored on BUILD 311-313 (two on an Apple TV, under device conditions) and this benchmark ran on the persona-offline-bench branch’s sim, so sim drift between builds and device-vs-soak differences may account for part of it. support:strong is the exception (archive 192, benchmark 276) and the noisiest (sd 224 s). tank:median is an elite on 3 samples.
  6. Roles are mostly honest at strong, loose elsewhere. tank:strong played tank 19/20 and utility:strong utility 14/20, but support:median / support:weak elites often played damage, and the weak/median tank and utility elites split roughly half-and-half between tank and utility.

Caveats

Section titled “Caveats”
  • 20 seeds per arm; survival sd is 40-80% of the mean, so differences under ~15 s (weak/median) or ~50 s (strong) are invisible at this n. Rerun with --n=50 for a firmer read on weak/median.
  • Seeds are consecutive integers (EvolveBatch.run_seed spacing); SeededRng hashes them, so that is fine, but every rerun uses the same 20, which is deliberate (comparable over time) and also means a quirk of these 20 maps is shared by every comparison.
  • One run was not scored: damage:strong on seed 1085003922 tripped SoakRun’s invariant check (tick 6241: pilot hp out of bounds: -0.011623/100), so damage:strong has n = 19. That is a real sim invariant failure on a deterministic seed (pilot HP slightly below 0), worth its own look; the learner drops such runs the same way.
  • Headless soak only; Apple TV/iPhone frame timing is not part of this.
  • Build drift: the archive’s elites were scored on BUILD 311-313; the benchmark flies them on this branch’s sim. Comparisons inside the benchmark (elite vs default, same sim, same seeds) are unaffected; comparisons against the archive’s recorded means are.

Reproduce

Section titled “Reproduce”
Terminal window
source ~/.secrets   # BULLET_HEAVEN_CONTROL_KEY
PERSONAS_API=https://bullet-heaven-control.chris-allen-06f.workers.dev node tools/persona_evolve/personas_http.mjs fetch archive.json
node tools/persona_evolve/benchmark/benchmark.mjs --archive=archive.json --n=20 --jobs=8 --out=results.json --md=table.md