A/B probe: paired bot-survival comparison

A/B probe: paired bot-survival comparison

scripts/ab-probe.sh balance --a <arm> --b <arm> runs the headless balance probe twice over the same seeds, once per arm, and says whether the bot survived longer, shorter or no differently under B than under A. It prints the paired 95% interval, the smallest difference that many seeds can resolve, and how many runs were cut off at the cap. The aim: a balance change or a config patch gets one number with an error bar before anyone argues about it.

It measures bot survival. NOT covered: human skill gap, feel and fun. The bot (SkilledAutoPlayer) is a proxy and is weaker than humans early on (skill bh-economy-sim). “Higher” means the bot lived longer, not that the game got better.

Status, 2026-10-22. On main with its probe side: dfr-probe-core (42f22c03) supplies the probe-result/1 file and --config=live|none, and scripts/gate.sh runs this tool’s offline tests (tools/ab/tests/fid_*.test.mjs, against a stub godot) in its fidelity step, which dfr-device-sim (a0421af2) added. Real runs: --a none --b live --seeds 24 (the sample below), an A/A of 8 seeds identical to the last digit, a rev:origin/main arm that matched the checkout seed for seed, an all-capped run that says “nothing was measured”, and a probe script error that stops in 1 s with Godot’s own text. Round 3 (after review): rev:0784003d --b tree had reported +248 s, p = 0.004, exit 0 because that commit predates the Difficulty core and played its own fixed difficulty while echoing normal. It is now refused before anything runs (see Conditions are checked, not trusted).

Quick start

# Does the live remote config change how long the bot survives? (A bare config means this checkout.)
scripts/ab-probe.sh balance --a none --b live

# Does this branch change it? A = origin/main as committed, B = this checkout including uncommitted edits.
scripts/ab-probe.sh balance --a rev:origin/main --b tree --seeds 60

# Shipped defaults against a candidate patch, keeping the evidence.
scripts/ab-probe.sh balance --a tree+none --b tree+patch:/tmp/candidate.json --out /tmp/ab-candidate

The difference is always B minus A, and higher survival is better. Put the baseline in A.

Arms

An arm is <code>[+<config>]. Either part may be left out: no config means live, and a bare config means tree. The report prints the full spelling, so none shows up as tree+none.

Part	Meaning
tree	This checkout, uncommitted changes included. The code part when only a config is given.
rev:<git ref>	A temporary detached worktree of that commit, with the current probe tooling (tools/probe_common, tools/balance_probe) laid over it before the import.
+live	The live remote config, as the probe resolves it. The config part when only a code is given.
+none	No remote config: the bible and tunables as the code ships them.
+patch:<file>	That remote-config patch file. A relative path is taken from your current directory.

Examples: tree (= tree+live), none (= tree+none), patch:p.json (= tree+patch:p.json), rev:origin/main+live, rev:a1b2c3d+patch:p.json.

live means what the probe resolves for the appstore channel at that tree’s BUILD. A rev: arm older than remote-config/ has no record at or below its BUILD, so live there is the bundled defaults, exactly like none. The probe reports that as source=none and still records the channel it looked in; the tool then runs the comparison and adds a WARNING saying so, instead of treating it as a failure. A live record that is a kill switch ({} or notice-only: every player on that build runs the bundled defaults) is compared too, with a WARNING that the arm is +none in all but name.

Conditions are checked, not trusted

Every arm must play exactly the difficulty, profile, hull, cap and start asked for. A rev: arm runs today’s probe tooling over old game code, and game code from before a condition existed parses the flag, echoes it and ignores it. rev:0784003d (2026-10-15, just before the Difficulty core 9b26e50a) against the checkout reported +248 s, p = 0.004: the whole effect was that commit’s fixed difficulty against Normal. So the tool checks twice:

1. Before anything runs, each arm’s own game code (the commit, read with git cat-file, or the checkout) must contain what applies each condition. The markers are capabilities in tools/ab/probes.json:

   Condition	Needs	Since
   --difficulty	sim/difficulty.gd, and tools/soak/soak_run.gd reading cfg.get("difficulty"	9b26e50a (2026-10-15)
   --profile	ai/skill_profile.gd, and profile_label in soak_run.gd	dd08f5a9 (2026-09-27)
   --meta=fresh	MetaRunSetup.apply in soak_run.gd	9891d3f6 (2026-09-27)

   An arm without them exits 2 naming the condition, the missing file and the commit that added it. That holds for --difficulty=hard too: such a tree has no difficulty to set. Compare older code by hand.

2. After every slice, the result’s echo (ship, difficulty, profile, ticks, start) must equal the request, and so must applied: the difficulty and profile the probe reads back from each finished run’s Sim / SoakRun (ProbeSession.applied_of), never from the flags. A missing, null or mixed value is refused as well.

--allow-mismatch (round 1 and 2) is gone: with every arm held to the request, two arms can no longer differ.

Options

Option	Default	Meaning
--seeds N	40	Paired seeds. Pick N from the table in How many seeds before you run. An explicit --seeds always wins over --detect, and warns when it is fewer than --detect needs.
--detect S	none	The smallest survival effect worth seeing, in seconds. Without --seeds the tool runs a pilot, then picks the seeds per arm (details).
--max-seeds N	64	Cap on the seeds --detect may choose.
--pilot N	8	Pilot seeds for --detect (reused as the first N seeds of the run; never above --max-seeds).
--opening	off	The change plausibly affects the first 25 s: print a caveat to compare the run’s early-death counts with the human anchor before trusting the bot result.
--seed0 S	5000	First seed. Both arms run S .. S+N-1.
--jobs J	4	Godot processes at once (at most 32). Each arm is split into J seed slices.
--difficulty D	normal	Difficulty the probe plays.
--profile P	median	Autopilot skill profile.
--ship H	cobalt	Hull.
--ticks T	21600	Run cap in sim ticks (21600 = 360 s). Runs still alive at the cap are censored.
--out DIR	none	Also write ab-result.json, arm-A.json and arm-B.json there.
--fail-on-worse	off	Exit 1 when the verdict is B WORSE.

--key value and --key=value both work. The only probe today is balance; feel is listed but not wired in yet (see Follow-ups).

Environment: GODOT (binary, default godot), AB_SLICE_TIMEOUT_S and AB_IMPORT_TIMEOUT_S (wall limits, below), AB_PROBE_TREE (a checkout other than this repo; used by the tests).

Exit codes: 0 a verdict whose 95% CI excludes 0 (B BETTER, or B WORSE without --fail-on-worse); 1 B WORSE with --fail-on-worse; 2 a harness or provenance error, nothing is concluded; 3 INCONCLUSIVE (the CI includes 0, or nothing was measured). 3 is not a pass: a script that gates on this tool must not read it as “no regression” (before 2026-10-22 an inconclusive run exited 0, which did exactly that); 128 + n when signal n stopped it: 130 Ctrl-C, 129 a closed terminal or dropped SSH session (SIGHUP), 143 SIGTERM, 131 SIGQUIT. In every one of those cases every godot process is killed, project.godot and the bus layout are put back and the temporary worktree is removed. SIGKILL cannot be handled: it leaves the godot processes to run to their wall limit, project.godot rewritten and the temporary directory (bh-ab-* under $TMPDIR, a registered worktree for a rev: arm) behind. Clean that up with git worktree remove --force <dir> and git checkout -- project.godot audio/bus_layout.tres.

What a run does

1. Checks everything it can before starting a process: arguments, godot and perl present, the patch file exists, each rev: ref is a commit, and each arm’s game code can apply every requested condition (Conditions are checked, not trusted).
2. For each rev: commit, git worktree add --detach into a temporary directory (hooks disabled, stale registrations pruned first) and lays the current probe tooling over it, before the import, so Godot’s class cache sees the same probe code in both arms.
3. godot --headless --path <tree> --import once per tree. The import rewrites project.godot and audio/bus_layout.tres; the tool saves both byte for byte first and restores them afterwards, even on Ctrl-C or a closed terminal. It never runs git checkout, so your own uncommitted edits to those two files survive.
4. Splits each arm’s seeds into --jobs contiguous slices and runs them, A and B interleaved, at most --jobs at a time: godot --headless --path <tree> -s res://tools/balance_probe/balance_probe.gd -- --seeds=N --seed0=S --ticks=T --profile=P --difficulty=D --ship=H --meta=fresh --config=<live|none|file> --json=<file> behind perl -e 'alarm shift; exec @ARGV'. The wall limit per slice is 300 + seeds_in_slice x max(60, ticks / 120) seconds (imports: 1800 s). It guards a hung process, not the sim speed; raise it with the env vars. A slice that prints a SCRIPT ERROR is stopped at once instead (see What makes it refuse), so a broken probe fails in seconds, not at the wall limit.
5. Validates every result file (What makes it refuse), merges the slices, pairs the arms by seed and computes the statistics.
6. Cleans up: temporary worktrees removed, the stray-worktree list pruned, children killed.

Leave the checkout alone while a tree arm runs. Every godot process reads the files as they are at that moment, so an edit between two slices mixes two versions of the code in one arm. To keep editing, commit and use rev:<commit> for that arm, which is a frozen copy.

Reading the report

This is a real run: scripts/ab-probe.sh balance --a none --b live --seeds 24 --jobs 4 on 2026-10-22, against dfr-probe-core at 087dfdf1 (BUILD 336). It took 222 s of wall time.

A/B probe: balance (difference = B minus A, paired by seed; higher survival is better)
  A  tree+none  git_rev=087dfdf1  build=336  source=none  config_id=default
  B  tree+live  git_rev=087dfdf1  build=336  source=live  config_id=b335-001
     path=remote-config/build-335.json
conditions: normal | median | cobalt | start fresh
seeds 5000..5023 (n=24 paired), cap 360 s (21600 ticks)

metric                                A        B      B-A       95% CI (paired t)       p
restricted-mean survival (s)      276.3    304.6    +28.3           +2.2 .. +54.4   0.035
  paired sd 61.8 s, SE 12.6 s, df 23; percentile-bootstrap CI +5.4 .. +55.2 (secondary, never decides)
median of paired differences +20.5 s, bootstrap 95% CI -6.3 .. +61.5, ties 0/24
sign test: B longer 15, B shorter 9, ties 0 (n_eff 24), exact p 0.307
dead before 30 s: A 0/24, B 0/24, B-A 0.0 points, exact McNemar p 1.000
boss seen: A 21/24, B 24/24, B-A +12.5 points, exact McNemar p 0.250
alive at the cap (right-censored at 360 s): A 0, B 3, both 0

VERDICT: B BETTER: restricted mean +28.3 s, 95% CI +2.2 .. +54.4 excludes 0
  at n=24 you can only see differences >= 35.3 s (80% power, 5% two-sided, normal approximation)
  note: the observed difference (+28.3 s) is smaller than the minimum detectable effect (35.3 s at n=24): a
        difference this size is found in fewer than 80% of runs, and a hit this small tends to overstate the
        effect; replicate on a fresh --seed0 block before acting on it
NOT covered: human skill gap, feel and fun (the bot is a proxy, weaker than humans early on)

How to read this one: the paired-t interval excludes 0, so the verdict is “higher”. The effect is smaller than the MDE, so the note says to replicate. The sign test (15 of 24 longer) and the median interval (-6.3 .. +61.5) do not back it up. That is a borderline result, not a settled one.

• Arm lines show what actually ran: the code revision, the build, the config source and its id. A tree arm with uncommitted changes is marked dirty (its git_rev is only the last commit).
• Restricted-mean survival is the mean of the restricted time per seed: a death counts as its time (at most the cap), a run still alive at the cap counts as exactly the cap. Such a run is right-censored there, and “at least the cap” is all that is known about it. This is the headline number. (A survivor’s t is a sum of ticks and carries float noise, 359.99999999998 to 360.000000000001, so it is taken as the cap itself; otherwise two survivors would differ by 2e-11 s and show up as a fake win or loss in the sign test.)
• The paired-t interval decides the verdict (df = n - 1). If it excludes 0 the verdict is “B BETTER” / “B WORSE”, otherwise “INCONCLUSIVE”. The tool never prints “no difference” or “no effect”: an INCONCLUSIVE verdict says what this n could and could not see (next two bullets).
• Percentile bootstrap (2000 resamples, fixed seed 0xB007, so the same data prints the same interval) is shown for the mean and for the median, and never decides. See the calibration for why.
• Median of the paired differences is printed with its tie count, and the ties are split by cause: both capped (alive at the cap in both arms, so nothing was measured for that seed, and a longer --ticks would measure it) and identical deaths (both arms died at the same instant, an identical trajectory). Above 50% ties the median interval is flagged and a note says so.
• Sign test: ties are dropped, n_eff is the number that count. Dead before 30 s and boss seen are paired proportions with an exact McNemar test on the discordant pairs. All three are descriptive.
• Alive at the cap counts the censored runs per arm and in both arms.
• The MDE line is the smallest true difference this n resolves with 80% power: 2.8 x SD_diff / sqrt(n). It is the normal approximation (the label says so). The t-based figure is in the JSON; at n = 8 it is about 17% larger. With no spread at all (every paired difference the same) there is nothing to estimate it from, and the line says “undefined” instead of printing 0.0 s.
• INCONCLUSIVE adds two lines: the MDE at this n, and “to detect a T s effect at this spread you need about N seeds per arm; this run had n”. T is --detect, or 30 s when it was not given (the line says so). When the run already had the seeds T needs, the line is decided by the 95% CI against T, never by seeds run vs needed (an inconclusive mean can sit anywhere inside +-1.96 SE, so the CI’s upper bound can reach about 3.9 SE while the MDE is only 2.8 SE): either “the CI rules out effects of T s or more in either direction”, or “the CI still allows an effect of T s or more (up to +X s), so one that size is NOT ruled out”.
• Secondary metrics are printed above the verdict, each with its own paired-t interval and MDE, from fields the probe already writes: HP lost per minute in the first 60 s (lost60_ps), HP lost per minute 180-360 s (lost_late_ps, only seeds where both arms were alive past 180 s, so conditional on surviving that long), and damage dealt per second (total_dmg over the run’s seconds). Their spread is much smaller than survival’s, so they resolve small changes survival cannot, but they never decide the verdict, they are not corrected for looking at four metrics at once, and “HP lost” falling with survival falling is possible. Kills per minute and time to the first level-up are not written by the probe today (adding them is a probe_common change, see Follow-ups). The definitions are secondary in tools/ab/probes.json.
• The fidelity note (FIDELITY: bots are not players ...) is always printed and puts two measured numbers side by side: the human share from tools/feel_probe/human_anchors.json (27 of 62 Hard-era human runs, 44%, ended under 25 s on 2026-10-22; read from the file, never typed here) and THIS run’s own bot deaths under 25 s per arm with the difficulty and profile played. It does not claim what bots do in general: the repo’s own research has weak and median bots dying under 30 s on Hard. With --opening a CAVEAT: OPENING line is added. There is no automatic detection from a config diff.
• Notes appear when something deserves a second look: many ties (naming how many are capped runs, and advising a longer --ticks only when some are); the bootstrap and the t interval disagreeing; a verdict whose size is below the MDE (a small hit is likely overstated: replicate it on a fresh --seed0 block before acting).

Worked example

The sample above: +28.3 s with SE 12.6 s at n = 24 (SD of the paired difference 61.8 s). t = 28.3 / 12.6 = 2.246 on 23 df, so p = 0.035 and the 95% interval is 28.3 +/- 2.069 x 12.6 = +2.2 .. +54.4 s. It excludes 0, so the verdict is B BETTER. But the MDE is 2.8 x 12.6 = 35.3 s, larger than the effect: n = 24 can only reliably see differences above 35 s, so this is a borderline hit that probably overstates the effect. The tool says so in a note. At n = 8 the same effect would be invisible (MDE 61 s).

How the verdict is decided

• Metric: restricted-mean survival time up to the cap, B minus A, per seed, then averaged.

• Rule: the 95% paired-t interval of that mean difference excludes 0. Then the verdict is “B BETTER” (or “B WORSE”); otherwise it is “INCONCLUSIVE”. “Inconclusive” means the data cannot separate the arms at this n. It is not evidence that they are the same.

• Why a verdict is inconclusive is in stats.inconclusive_reason and in the verdict text. Zeros mean different things, and the report keeps them apart:

  Reason	What happened	What the report says
  ci_includes_zero	The ordinary case: the interval includes 0.	restricted mean +X s, 95% CI a .. b includes 0
  identical	Every difference is exactly 0 and no seed was alive at the cap in both arms. What two runs of the same arm must give, since the sim is deterministic.	the arms are identical on every paired seed (all N differences are exactly 0, none alive at the cap in both arms)
  all_capped	Every seed was alive at the cap in both arms. Nothing was measured: the arms are NOT shown to be the same.	all N paired seeds were alive at the cap in both arms, so nothing was measured; raise --ticks, no interval and no p value
  ties_only	Every difference is 0, some seeds capped in both arms and the rest identical deaths.	both counts, and a longer --ticks for the capped seeds
  n_lt_2	Fewer than 2 pairs: no interval exists.	n=1, an interval needs at least 2 paired seeds

Why the t interval decides and the bootstrap does not

With no real difference between the arms (A/A), a method that is right 95% of the time should still print a verdict about 5% of the time. This is how often each interval did, on a capped lognormal (cap 360 s, about 39% of runs censored), 10 000 experiments per cell:

n	paired-t interval (decides)	percentile bootstrap, mean	percentile bootstrap, median
8	4.8%	12.2%	2.8%
12	4.8%	9.5%	3.7%
16	4.8%	7.3%	1.4%
24	5.0%	7.2%	1.3%
40	5.0%	6.0%	0.6%
100	4.8%	5.4%	0.1%

The bootstrap of the mean is anti-conservative at small n (12% at n = 8). The bootstrap of the median is the opposite: ties at the cap make it conservative, so it rarely fires at all, which is lost power and not extra safety. Only the t interval is calibrated at every n, so it decides and the others are printed beside it. tools/ab/tests/fid_ab_stats.test.mjs keeps a smaller version of this simulation (500 experiments, at most 8% at n = 8, 24 and 40).

How many seeds

Sizing a run with --detect

--detect=28 says “I want to be able to see a 28 s change”. With no --seeds the tool (1) runs a pilot of --pilot seeds (8) in both arms, (2) takes the pilot’s paired spread and pools it with a stored spread of 58 s worth 4 degrees of freedom (PRIOR_SD_DIFF_S, PRIOR_DF in ab_stats.mjs; derived from the measured MDE of about 33 s at 24 seeds; the pilot has 7 df, so it outweighs the prior), (3) picks ceil((2.8 x sd / S)^2) seeds per arm, never fewer than the pilot and never more than --max-seeds (64), and (4) runs only the missing seeds, so no seed is run twice. The formula is the normal approximation, a few seeds below a t-based figure at small n; the finished run prints its real MDE. The choice is printed to stderr as it is made and again in the report (seed count: ...); when the cap bites the report says so.

The stored spread was measured for Normal, the median profile and the 360 s cap. Under any other --difficulty, --profile or --ticks it is not used (priorFor returns null): the pilot alone sizes the run, and --seeds with --detect cannot be judged short. If the pilot has no spread at all (every pair tied, usually every run alive at the cap, which says nothing about the real spread) the stored spread sizes the run alone, or, with no stored spread, the default 40 seeds. Only the pilot’s spread sets n, never its mean. With an explicit --seeds there is no pilot: the report shows what --detect would have needed and a WARNING when --seeds is below it. --pilot and --max-seeds without --detect are ignored with a WARNING. At the stored spread: 28 s needs 34 seeds per arm, 20 s needs 66, 10 s needs 264 (past the cap: raise --max-seeds knowingly, or accept a wider MDE).

The spread of the paired difference decides everything, and it is the number to read off your own report (paired sd). The table assumes it is near 58 s: a per-arm SD near 41 s with the two arms sharing little beyond the seed, which is what the 40-seed intervals in the 2026-10-20 balance note imply (its +49 s [+34, +70] is about +/-18 s wide, and 58 / sqrt(40) x 2.02 = 18.5).

n	SE (s)	MDE, normal (s)	MDE, t-based (s)	P(verdict “higher”) for a true +30 s
8	20.5	57	67	24%
12	16.7	47	52	38%
24	11.8	33	35	68%
40	9.2	26	26	89%
100	5.8	16	16	>99%

Other true shifts at n = 40: +10 s is found 19% of the time, +20 s 55%, +45 s 100%. Seeds needed for an MDE of 30 s / 20 s / 15 s / 10 s: 30 / 66 / 118 / 264. (Power is for an uncensored Gaussian difference. Heavy capping shrinks the spread, so a capped run is somewhat easier to read; your report shows the real SD.)

The old rule of thumb was too optimistic. The earlier research notes call a difference under about +/-10 s noise at 40 seeds (docs/research/2026-10-16-difficulty-normal-calibration.md, docs/research/2026-10-20-balance-evidence-hulls-crit-early-run.md). At 40 seeds the numbers above give an SE near 9 s, a 95% half-width near 18 s and an MDE near 26 s. That note’s own tables agree: its 40-seed median differences come with intervals such as +27 s [+9, +59] and +10 s [-8, +48]. A true 10 s change is found about one time in five at 40 seeds.

Decide n before you run. If 40 seeds are inconclusive, do not add 40 more and test again until it says “higher”: repeated looks inflate the false-positive rate. Plan the larger n up front, or replicate on a fresh block (--seed0 5040) and treat it as a second experiment. One verdict is one test at 5%: comparing 15 pairs of hulls and acting on the “wins” will produce a false win now and then.

What makes it refuse

Exit 2, no verdict, a one-line reason. The tool stops at the first failure and kills the other runs.

• Before anything starts: unknown probe or option, bad arm, a disabled probe, godot or perl missing, a patch file that does not exist, a rev: ref that is not a commit, a ref beginning with -, an arm whose game code predates a requested condition (capabilities in probes.json).
• During: a slice or an import over its wall limit (the message ends with the last stderr lines); SCRIPT ERROR on a slice’s stderr; no result file (the message says how godot ended, and that a probe from before dfr-probe-core cannot take --config=live|none); a result that is not JSON. A SCRIPT ERROR stops the slice at once: a script error inside a SceneTree script’s _init aborts _init, so quit() is never reached and headless Godot idles forever instead of exiting. The tool kills the slice once the error line and its three context lines have arrived (or two seconds after the line, when fewer follow) and reports the error text, and it checks for the error before it checks the wall limit, so a slice that hit one is never reported as “timed out, raise AB_SLICE_TIMEOUT_S”. A SCRIPT ERROR printed by godot --import (a stale class cache prints harmless ones) is not an error.
• In a result file: schema missing or not probe-result/1; status other than pass (the probe’s own invariant failures and config problems are printed); a pass that still lists failures; a result from a different tool; missing git_rev, ship, difficulty, profile, start or config.source; seed0, seeds, ship, difficulty, profile, ticks or start not equal to the request; no applied record, or an applied.difficulty / applied.profile (read back from the Sim) that is missing, null, mixed or not the one asked for; cap_s disagreeing with ticks / 60; the wrong number of rows, a duplicate or a missing seed; a row with dead: false earlier than the cap (the harness ended the run early, so it is not a censored survivor); a config.source that is not the config asked for (--config=live may come back as none only when the probe also reports the channel it consulted: that is “no record”, accepted with a warning, see Arms).
• Across slices and arms: slices of one arm disagreeing on build, revision, start, hull, difficulty, profile, applied, ticks, cap or config; the two arms disagreeing on difficulty, profile, hull, ticks, cap, seeds or start (a backstop: each slice is already held to the request).

How a godot process ended never decides by itself; the result file does. After a valid file, a nonzero exit or a signal is only a warning, and headless Godot’s known teardown abort (exit 134 as a shell sees it, SIGABRT as node sees it; docs/architecture/godot-gotchas.md) is silent. The same goes for --import, which only fails on a timeout.

Files written with --out

• ab-result.json, schema ab-result/1: probe, n, cap_s; arms.A / arms.B (the arm spec, code, ref, commit, config, dirty, overlay, the probe’s git_rev, build, config_info {source, path, config_id, and channel / kill_switch when the probe reports them}, difficulty, profile, ship, start, applied, ticks, cap_s, seed0, seeds, slices with their wall time); provenance {match, mismatches}; stats (everything in the report, unrounded); verdict (better / worse / inconclusive), verdict_label (B BETTER / B WORSE / INCONCLUSIVE), needed_line, seed_plan, opening, human_anchors; stats.secondary and stats.detect; verdict_text, mde_line, notes, warnings; timing; pairs (the per-seed rows of both arms). The printed report is a pure function of this file.
• arm-A.json, arm-B.json: each arm’s slices merged into one clean probe-result/1 document. They keep the fields this tool reads and drop the rest of what the probe wrote: git_dirty, and any row field other than seed, t, dead, cause, level, boss_seen. They are a record of what was compared, not a copy of the probe’s raw files.

What the tool needs from the probe

Checked on 2026-10-22 against main (tools/probe_common/probe_session.gd, probe_args.gd, live_config.gd), by running the tool against it, not only by reading it.

• --config=live|none|<absolute file> and --json=<file> on the command line shown above.
• A probe-result/1 file with: schema, tool (balance_probe), status (pass or fail), invariant_failures[], config_problems[], build, git_rev (may be ""), start (fresh), ship, difficulty, profile, applied {difficulty, profile} read back from the runs, config {source, path, config_id, channel, kill_switch}, seed0, seeds, ticks and / or cap_s, and rows[] of {seed, t in seconds, dead, and optionally cause, level, boss_seen}.
• A survivor has dead: false and t equal to the cap to within float noise (the real probe sums ticks: 359.99999999998 to 360.000000000001 at the 360 s cap, inside the one-tick tolerance of the early-end check).
• config.source echoes the config that was applied: live, none, or file for a patch. config.channel is non-empty only when --config=live was consulted, and that is the only way the tool can tell “live found no record” (source: none plus a channel, accepted with a warning) from “this probe never looked at live” (source: none and no channel, refused).

Limits

• It measures the bot. Human skill, feel and fun are outside it, and the bot is weaker than humans early on, so an effect that matters in the first minutes can be larger or smaller for people.
• Fresh-save starts only (--meta=fresh), one hull, profile and difficulty per run. Effects that need shop progression, or that only show on another hull, need their own runs with --ship / --profile / --difficulty.
• The seed pairs the starting conditions, not the trajectory. Once the arms diverge they are different games, so pairing probably buys little [Likely]; read the real paired sd rather than assuming a gain.
• rev: arms run old game code under today’s probe tooling. A commit that predates a requested condition is refused before it runs (today: anything before 9b26e50a, 2026-10-15, for --difficulty). If the probe calls a sim API the old commit lacks, godot prints a SCRIPT ERROR, the tool stops that slice within a couple of seconds and exits 2 with the error text. Nothing under sim/ is ever overlaid.
• The cap hides anything that happens after it. If many seeds sit at the cap, raise --ticks.
• The feel probe is not wired in (see Options).

Tests

node --test tools/ab/tests/fid_*.test.mjs

No Godot, no network, under 10 seconds. fid_ab_stats.test.mjs pins the statistics against numbers computed independently (scipy / by hand) and runs the A/A and power simulations. ab_stats.mjs imports nothing: the t distribution (pairedT, tCrit95, tTwoSidedP, meanSd) is a verbatim copy of tools/persona_evolve/benchmark/benchmark.mjs, so a refactor of that file cannot change an A/B verdict. A parity test compares the copy with the original and flags drift (it skips itself if that file moves). fid_ab_probe.test.mjs drives the real CLI against tools/ab/tests/stub_probe.mjs, a fake godot (selected with GODOT) whose runs are a pure function of seed and config. The stub injects the probe-side faults behind most of the refusals above (bad schema, early end, short slice, duplicate seed, SCRIPT ERROR buried in a flood, a SCRIPT ERROR followed by an idle process the way real Godot behaves, exits and signals, hangs, a live config with no record, a kill switch, a probe that echoes one difficulty and plays another, a commit from before the Difficulty core). It is a copy of the contract above, not the contract itself, so it can drift from the real probe (see Follow-ups). The probe side of applied is pinned in Godot by tests/test_probe_start_parity.gd. scripts/gate.sh runs these files in its fidelity step (the tools/*/tests/fid_*.test.mjs glob).

Follow-ups

1. Determinism check after any change to the probe or the sim path: scripts/ab-probe.sh balance --a none --b none --seeds 8 --jobs 8 --out /tmp/ab-aa. The sim is deterministic, so the verdict must be “the arms are identical on every paired seed” with alive at the cap ... both 0. On 2026-10-22 it was (56 s; mean 299.3 s in both arms). When seeds sit at the cap in both arms (a strong profile, a short --ticks), every difference is trivially 0 and the tool says “nothing was measured” or names the capped seeds instead of “identical”; raise --ticks until none are. If the arms differ, the probe is not deterministic and every other number from this tool is suspect.
2. Commit a real probe-result/1 file from main as a fixture and validate the stub’s output against it, so the stub cannot drift from the probe.
3. Point docs/reference/running-and-testing.md and the skills bh-feel-probe and bh-economy-sim at this tool, and correct the “+/-10 s is noise at n = 40” lines in the two research notes named above.
4. When a new condition lands in the game (a flag the probe passes that old code would ignore), add its markers to capabilities and, if the Sim can report it, to applied in probes.json and ProbeSession.applied_of.
5. Kills per minute and time to the first level-up as secondary metrics: the balance probe does not write them (the feel probe computes the level-up times). Add them to the row in tools/balance_probe/balance_probe.gd (or probe_common) and to secondary in probes.json; the report picks them up with no other change.
6. The bot-vs-human KS distances the fidelity note points at are not in this repo as numbers; the note links the feel-metrics doc where the human anchors live. Cite the KS source there when it is written down.
7. Wire in the feel probe: since feel-metrics-v2 it writes per-seed rows (seed, t, dead); it needs its probes.json entry (script, overlay, capabilities) and a real-run check.