Skip to content
Dark Cosmos — Dev Docs

Fidelity probe core

Fidelity probe core: probes that play like a new player

Section titled “Fidelity probe core: probes that play like a new player”

tools/balance_probe/balance_probe.gd and tools/feel_probe/feel_probe.gd (wrappers scripts/balance-probe.sh and scripts/feel-probe.sh) share one start, one config path and one result contract, all in tools/probe_common/. Added 2026-10-22 (task dfr-probe-core). The probes are dev tooling: nothing under /sim or in the shipped game references them.

Why

Section titled “Why”

Until this change both probes measured a game no store player runs, and nothing in their output said so.

Before Effect
SoakRun was given no shop save, so no MetaRunSetup ran and locked_weapons stayed {} all 15 weapons were on offer from level 1; a fresh save has 13 of them behind Arsenal unlocks, so a fresh-save run only ever held blaster, orbit and nova
bundled defaults, unless a --config file was passed store players run the bundled defaults plus the patch the Worker serves for their BUILD (1.1: the Orbit/Nova patch)
balance_probe defaulted to --difficulty=hard a new player is on Normal
unknown flags were ignored, numbers read with int("abc") (= 0) --ship did nothing for months; --seeds=abc ran zero seeds and printed an empty summary
quit(0) whatever happened; feel-probe.sh turned exit 134 into success invariant failures and crashes read as green
no provenance in the output which build, commit and patch produced a number was unknowable
patches went in through Tunables.apply and were never undone a run left the process tuned

What a default run is now

Section titled “What a default run is now”
  • Start: a brand-new save (--meta=fresh): a new MetaState per seed through MetaRunSetup.apply, the same call main.gd and soak make. --ship picks the hull (default cobalt); the hull does not have to be unlocked.
  • Patch: --config=live is the record the Worker would serve this BUILD on --channel (default appstore), applied the way the game applies a download (Tunables.apply, then GameContent.load_db for the bible overlay).
  • Difficulty: Normal. Hard stays one flag away.
  • Honest results: unknown flags and bad values are refused, a provenance line is printed first, and the exit code and the --json file say whether the run was clean.
  • No leaks: Tunables are snapshotted before the config is applied, put back between seeds, and restored when the probe finishes.

The first line the probe prints after Godot’s own banner:

[provenance] feel_probe build=336 rev=7309f2b0 start=fresh ship=cobalt difficulty=normal profile=median config=live b335-001 [remote-config/build-335.json, resolved build 335, channel appstore] seeds=8@5000 cap=300s

(dirty tree) follows the revision when tracked files differ from HEAD, (no git) replaces it outside a checkout, and start=none LEGACY (no shop save: ...) is spelled out so an old-style run cannot be mistaken for a new-player one. config=none (config_id default) is --config=none; a --config=live that matched no record says so and names the channel and build, and a {} record reads kill switch, because both mean bundled defaults ran. +tun, +bib or +force after the config says overrides were layered on top.

Flags

Section titled “Flags”

Both probes refuse any flag not listed here (exit 2, with the valid list), and a flag given twice.

Flag Default Meaning
--seeds=N balance 20, feel 8 number of seeds, at least 1
--seed0=N 5000 first seed; seed i is seed0 + i
--ticks=N balance 21600, feel 18000 per-run cap in 60 Hz ticks (360 s, 300 s)
--profile=P balance weak, feel median weak, median, strong (ai/skill_profile.gd) or off (plain SkilledAutoPlayer)
--difficulty=D normal hard or normal; anything else is refused (the game maps unknown ids to Normal)
--meta=M fresh fresh = new save through MetaRunSetup; none = the legacy start (every weapon open, no hull or meta setup)
--config=C live live = the checked-in record for this BUILD (below); none = bundled defaults; or the path of a patch .json
--channel=X appstore channel --config=live resolves for, [a-z]{1,12}
--ship=S cobalt hull id from ShipBonuses.TABLE
--json=PATH none write the probe-result/1 envelope; any stale file at PATH is deleted first

balance_probe only: --tun=KEY=V[,..] and --bib=KEY=V[,..] (tunable and bible overrides on top of the config), --start=, --grant_at=T:id[+id], --ramp=KEY:from:to:t_end[;..], --watch=orbit,nova, --pilot_hp=N, --label=NAME. A malformed override or ramp is a usage error, not a silent zero.

feel_probe only: --force=Class.NAME=V[,..] (set a registered tunable outside its remote-config bounds, for calibration such as the App Clip’s 5x XP; never part of a patch) and --no-scan (skip the static snap-vs-ramp scan).

--config=live: which record

Section titled “--config=live: which record”

tools/probe_common/live_config.gd mirrors resolveConfig in telemetry/src/worker.js over the files in remote-config/:

  1. The record for the exact BUILD, build-N@channel.json first, then build-N.json; otherwise the nearest OLDER build’s record, same order. Never a newer one.
  2. The repo stores every patch as build-N.json whatever key it was published to, so the channel is read from the marker in the record’s config_id (b331-dev-001 is dev; a plain b335-001 is the plain key every channel sees). Known markers: dev, web, testflight, appstore (a test pins the list to scripts/publish-config.mjs). So --channel=appstore never picks up a dev-only record.
  3. A {} record (or one holding only a notice) is a kill switch: the search stops and the bundled defaults run, config_id is default.
  4. A record the game would reject (RemoteConfigState.valid) or that is not a JSON object is reported as a config problem; no patch applies, as on a client.
  5. No record at all (or a missing remote-config/ directory) applies nothing and reads source: none.

This reads the CHECKED-IN records, not the Worker’s KV. A record that was never published, or was later unpublished, still reads as live here; check telemetry/deploy.sh --check-only and the console /config before quoting a number as “what players run”.

The probe-result/1 envelope

Section titled “The probe-result/1 envelope”

With --json, the probe writes this object. Every key the tools wrote before is still there with the same value (the contract fields are written last, so a tool key cannot turn a failing envelope into a passing one).

Field Meaning
schema probe-result/1
tool balance_probe or feel_probe
status pass or fail. fail means an invariant failure or a config problem. A feel band that reads FAIL is a finding, not a failed run.
build Sim_Const.BUILD of the checkout that ran
git_rev short HEAD hash, empty when git is unavailable
start fresh or none
ship, difficulty, profile the flags as parsed (an echo: a tree that predates a condition still echoes it)
applied.difficulty, applied.profile what the runs actually played, read back from each finished SoakRun and its Sim (ProbeSession.applied_of), not from the flags: null when the tree cannot say (no sim/difficulty.gd), mixed when seeds disagree, profile off when no knobs drove the autopilot
config.source live, file or none (--config=none, or live with no record)
config.path repo-relative record, the --config file, or empty
config.config_id the patch’s id; default for no patch or a kill switch; file for a file that names none
config.channel the channel consulted (live only, else empty)
config.resolved_build the build whose record was picked, 0 for none
config.kill_switch true when live resolved to a {} (or notice-only) record: every player on that build runs the bundled defaults
seed0, seeds, ticks as run
cap_s ticks / 60
invariant_failures seed N: ... per SoakRun invariant failure
config_problems patch or override problems (Tunables.apply, GameContent.load_db, record validation)

Extras beyond the contract: git_dirty (tracked files differ from HEAD), locked_weapons (weapons behind an unlock at the first run start: 13 fresh, 0 legacy), config.overrides (which of tun, bib, force were layered on). Tool payload: balance adds label, patch, rows; feel adds summary, rows, scan, patch.

Exit codes

Section titled “Exit codes”
Code Meaning
0 clean pass
1 invariant failures or config problems. The run still completes and writes the file; the problem is also on stderr before the first seed.
2 bad invocation: unknown or repeated flag, bad value, unreadable --config file, malformed --tun/--bib/--force/--grant_at/--ramp, unwritable --json. Usage is printed, nothing runs.

Godot 4.7.1 exits 0 at headless teardown, so the old “134 is the known SIGABRT” rule is gone; 128 and above is a crash.

scripts/balance-probe.sh and scripts/feel-probe.sh always pass --json (yours, or a throwaway path), delete any stale file there first, and exit 0 only when godot exited 0 and the file parses as a probe-result/1 with status: pass (python3 -I). A SCRIPT ERROR, a crash, a missing or unparseable file, status: fail and the probe’s own exit 2 are all exit 1 from the wrapper. Every ERROR and invariant line is echoed in full; only the per-seed progress is tailed. A relative --json path resolves against the repo root.

Reproducing the old numbers

Section titled “Reproducing the old numbers”
Previous behaviour Flags now
balance_probe defaults (Hard, every weapon open, bundled defaults) --meta=none --config=none --difficulty=hard
feel_probe defaults --meta=none --config=none
either, at Normal add --difficulty=normal
either with --config=patch.json --meta=none --config=patch.json

Checked on 2026-10-22 against origin/main (7309f2b0): with --meta=none --config=none --difficulty=normal --profile=median on seeds 5000-5002, every old top-level key of both probes’ JSON is byte-identical (the new keys are additive), and the stdout differs only by the added [provenance] lines. Numbers quoted in earlier research docs came from the legacy start unless they say otherwise.

Legacy against new

Section titled “Legacy against new”

Normal difficulty, median profile, cobalt, 21600-tick cap (360 s), measured on 7309f2b0 (BUILD 336, live record b335-001). Survival is the probe’s own nearest-rank median with p25 and p75, in seconds.

Arm (flags) 8 seeds (5000-5007) 24 seeds (5000-5023) Boss seen (24) Weapons held at the end (24 runs)
A legacy: --meta=none --config=none 309.3 (294.1 / 310.1) 297.5 (275.4 / 328.3) 24/24 15 distinct: blaster 24, beam 7, scythe 6, scar_tissue 5, turret 5, blade 4, momentum_core 4, railgun 4, scatter 3, chalice 2, nova 2, pulse 2, thunder_wake 2, chronowell 1, orbit 1
B fresh start only: --meta=fresh --config=none 308.4 (269.8 / 328.2) 286.8 (265.1 / 299.6) 21/24 3: blaster 24, orbit 22, nova 21
C live patch only: --meta=none --config=live 309.3 (294.1 / 310.1) 297.5 (275.4 / 321.4) 24/24 as A
D new default: --meta=fresh --config=live 328.4 (280.4 / 340.1) 308.3 (276.3 / 328.4) 24/24 3: blaster 24, orbit 22, nova 21
E old balance default: --meta=none --config=none --difficulty=hard 69.2 (26.7 / 84.0) 59.8 (50.0 / 84.0) 0/24 14 distinct

What it says:

  • The pool is the big change. A fresh save can only ever hold blaster, orbit and nova; the legacy probe handed out railgun, blade, scythe and ten more. Survival medians of A and D sit within 10 to 20 s of each other, well inside the 50 s p25-p75 spread, so the headline is not a number: the legacy probe measured a different game.
  • The live patch only shows up on a fresh start. The Orbit/Nova patch moves D against B by +21.5 s median (p25 +11.2, p75 +28.8) but A against C by 0.0 s, because the legacy autopilot held orbit in 1 and nova in 2 of 24 runs. A patch to those weapons measured on the legacy start would have looked like a no-op.
  • The old Hard default was nothing like a new player’s game: median about a minute, the boss never seen.
  • 8 seeds and 24 seeds agree on direction, not on size; quote 24.
  • An actual 8-seed run of arms A and D returns rows identical to the first 8 rows of the 24-seed runs (each seed’s run depends on its seed alone).

Feel probe, 8 seeds, same flags: legacy (A) died in 4/8 with 24.9% of damage from the blaster and 11 weapons in play; the new default (D) died in 3/8 with the blaster at 36.2%, orbit 9.6% and nova 6.8%. Dead air reads FAIL in both (0.45 and 0.42 of seconds).

What this does not tell you

Section titled “What this does not tell you”
  • The autopilot is a model of a player (weak, median, strong), not a person. Only a device shows touch handling, frame pacing, what the shop feels like and whether a run is fun.
  • --meta=fresh is a first-launch save. A player with unlocks, gold spent and a drone is not covered; there is no flag for a mid-game save yet.
  • Co-op and the tutorial are not played by these probes.
  • --config=live reads the repo, not the Worker’s KV (above).

Where the code is

Section titled “Where the code is”

tools/probe_common/: probe_args.gd (flag specs and parser), probe_start.gd (the SoakRun config), live_config.gd (record resolution), probe_report.gd (envelope, banner, exit codes, git provenance), probe_session.gd (one invocation: open, begin_seed, note_run, finish). A new flag goes in ProbeArgs and in the whitelist test tests/test_probe_args.gd. Tests: tests/test_probe_args.gd, test_probe_report.gd, test_probe_live_config.gd, test_probe_start_parity.gd (against the MetaRunSetup oracle), test_probe_cli.gd (child-process runs) and scripts/test/probe-wrapper.test.mjs (the wrappers, with a fake godot).