Fidelity: release-VM smoke

Fidelity: release-VM smoke

scripts/release-smoke.sh boots the exact candidate under the macOS release export template, the GDScript VM that devices run, and takes the device-only auto-arm path with the network off. It answers one question: does this commit launch on a release build, or does it crash before the first frame?

Why it exists

scripts/gate.sh boots the game twice (the “boot smoke” and the “device-boot smoke” steps), both in the editor’s debug VM. The two VMs disagree about errors. The debug VM reports a bad call as a soft SCRIPT ERROR and carries on. The release VM compiles those checks out, so the same call dereferences null and the process dies with a segfault.

On 2026-10-09 the boot /status read debug_settings before it existed. The debug VM logged a SCRIPT ERROR and kept going; the release VM segfaulted at launch on Apple TV and iPhone. The gate has since gained the device-boot step, which takes the same device-only boot path and fails on a SCRIPT ERROR in the debug VM’s log, so it catches that exact line today. What it cannot see is anything that behaves differently in the release VM or in an exported pck (a different file set from the source tree). This smoke is that second opinion, run on the pck itself.

What it covers, and what it does not

Covers:

• The commit’s scripts and scenes, exported with the tracked platform/tvos presets, running under the stock macOS release template: a crash, a hang, a script parse error, or (Store pass) the boot code failing to reach its first save file.
• The device auto-arm boot path (ControlClient armed, /caps and /status built, a poll queued) in the release VM, with the network off: the marker DEVICE_BOOT_SMOKE armed=true captured=N http_requests=0 is required.
• Files the export filter strips (tests/*, tools/*, telemetry/* and so on). A boot path that preloads or load()s one works from source and is missing from the pck.

Does not cover (the verdict says so in its caveat):

• The device GPU and Metal. The run is headless, so nothing is rendered. A shader, driver or Metal fault is invisible.
• The Godot 4.8 P3 fork that exports the iOS and tvOS candidates. This is the stock 4.7.1 release VM, not the fork’s. A fork-only crash (the GPUFamilyApple6 gate that rejected build 18 is the precedent) is out of reach. Fork fidelity still needs a device run.
• Native iOS and tvOS plugins (DcHaptics, Sentry), on-device state, memory limits, and anything a macOS host cannot reproduce.
• Soft script errors in the release VM. They are compiled out, so only crashes, hangs and load errors show. Keep the gate’s debug-VM SCRIPT ERROR check; this smoke does not replace it.
• A windowed run. See “Why there is no –windowed” below.
• Anything the boot does not reach: a defect behind a menu, a run or a boss fight is invisible to it.

Run it

scripts/release-smoke.sh                      # both passes, about 25 s on a warm machine
scripts/release-smoke.sh --dev                # Dev pass only
scripts/release-smoke.sh --store              # Store pass only
scripts/release-smoke.sh --pck FILE [--store] # smoke a pck you already have, no export
scripts/release-smoke.sh --secs 60 --keep     # longer guard; keep the scratch dir and print its path
scripts/release-smoke.sh --result out.json    # also write a fidelity-result/1 JSON (run logs are copied beside it)
scripts/release-smoke.sh --classify LOG EXIT  # classify a finished run, pure: no Godot, no export

Run it before a store or TestFlight submission, and after touching boot code (main._ready, autoloads, ControlClient, BuildConfig, MetaStore). It is not wired into scripts/release/release.sh yet: that is a release-process change and needs Chris’s OK. The natural place is just before the build step, as a hard stop on CRASHED, HUNG, SCRIPT-ERROR and MARKER-MISSING.

The device-sim harness picks it up from tools/device_sim/registry.d/release-smoke.json (headless tier, global scope, needs godot and macos-template). One mismatch to know about: the harness’s macos-template check also wants the gitignored root export_presets.cfg, which this script does not use (it exports from the tracked platform/tvos presets), so in a fresh worktree the harness can report SKIP for a tool that runs fine by hand.

What a run does

1. Finds the editor (godot, or BH_RELEASE_SMOKE_GODOT) and the matching template (below). It prints both versions.
2. For each pass, exports a pck (cold, BH_EXPORT_COLD=1) from the tracked platform/tvos/export_presets.cfg, through godot_export pack so stale caches and rewritten *.import files are handled:
   • Dev pass: preset iOS-Dev (custom features dev,dev_filelog), run with -- --device-boot-smoke.
   • Store pass: preset macOS (no dev feature, what scripts/release/build-macos.sh ships), run plainly.
3. Copies the template’s release binary to a distinct name, bh-release-smoke, with the pck beside it as bh-release-smoke.pck. A release binary refuses --main-pack and loads <binary>.pck from next to itself. The distinct name means a crash report is bh-release-smoke-<date>.ips, so it cannot be mistaken for the editor’s.
4. Runs it headless with --quit-after 600 under a wall-clock guard (--secs, default 60; a healthy boot takes about 5 s), with HOME set to a scratch directory, one per pass.
5. Restores project.godot and audio/bus_layout.tres if the Godot run rewrote them (and says so), and removes the scratch directory unless --keep.

The root export_presets.cfg is gitignored and absent from a fresh worktree, which is why the tracked platform/tvos presets are used.

Reading the verdict

The last line of stdout is the verdict. Exit codes:

Verdict	Exit	Meaning
PASS	0	Every pass booted. The Dev pass printed the marker; the Store pass wrote its own boot file.
CRASHED	1	The process died on a signal (exit 128 or more, e.g. 139 = SIGSEGV). Decided from the exit status, never the log.
HUNG	1	The wall-clock guard killed it (guard exit 124, or a raw SIGALRM 142). Never reported as a crash. A heavily loaded machine can be slow: re-run with a longer --secs before calling it a hang.
SCRIPT-ERROR	1	The log has SCRIPT ERROR or Parse Error. Parse errors are not compiled out, and the process still exits 0.
MARKER-MISSING	1	Exit was clean but the proof of boot is absent (see below).
SKIP	2	No macOS release template for this Godot. Nothing was checked; this is not a pass.
INCONCLUSIVE	2	Version skew, an export that failed, or a pck from another engine. Nothing was checked.
(usage)	64	Bad arguments.

Exit 2 is deliberately not 0: a missing template must never read as a pass. Every failing pass is reported; the verdict is the first failing pass’s.

For the device-sim runner the mapping is: PASS is pass; CRASHED, HUNG, SCRIPT-ERROR, MARKER-MISSING are fail; SKIP is skip; INCONCLUSIVE is inconclusive. The result JSON carries export_s, run_s, pck_mb, exit_code and captured for the Dev pass and the same names with a store_ prefix for the Store pass (no captured: it prints no marker), and its caveat restates what is not covered.

Why the proof differs per pass. The Dev pass must print DEVICE_BOOT_SMOKE armed=true captured=<n> http_requests=0 with n at least 1. A release binary block-buffers stdout when it is redirected, so the marker (and the banner) appear only after a normal exit: a segfault loses them. That is why a crash is read from the exit status. A store export ignores the flag by design, so it prints nothing. A pck whose main script fails to parse still exits 0, so exit 0 alone proves nothing. The Store pass therefore also requires that the game’s own boot code wrote a file (canary.txt or meta.json, from MetaStore in main._ready) under its scratch user directory. No file means MARKER-MISSING.

Bare ERROR: and WARNING: lines never fail a run. The macOS template has no dc_haptics GDExtension, so every run logs a few Can't open GDExtension dynamic library errors that are expected. They are hidden from the failure tail; the full log is kept with --keep or --result.

Engine pairing

A pck built by one engine and run by another looks exactly like a crash, so the script pairs them explicitly:

• The template is found by the exact godot --version string: ~/Library/Application Support/Godot/export_templates/ <version>/macos.zip, looked up under your real home (the child run gets a scratch one). The extracted binary’s --version must equal the editor’s, else exit 2 with both versions printed.
• A pck’s header records the exporting engine’s version (GDPC, pack format 4, then major, minor, patch). A --pck from a different engine than the template is INCONCLUSIVE and names both versions.
• --template ZIP (or BH_RELEASE_SMOKE_TEMPLATE) overrides the lookup.

To smoke the pck that build-macos.sh ships, point --pck at the .pck inside the built app’s Contents/Resources with --store. The script copies the template binary next to it, so it never runs the signed, sandboxed app itself.

Why there is no –windowed

A windowed run is a live platform. A fresh HOME mints an install id, and EventQueue and RemoteConfig then send telemetry and fetch remote config. The smoke flag silences only the ControlClient. The option was dropped rather than shipped unverified; --windowed exits 64 with this explanation. A headless run reaches every boot path that matters here, and EventQueue.push and the remote-config fetch are quiet headless.

The one code change: ControlClient.smoke_exported

The flag --device-boot-smoke used to be inert in any export (exported = OS.has_feature("template")), so the release VM could never be armed from a pck. net/control_client.gd now decides exported through a pure smoke_exported(template, dev), which is template and not dev. A store export (template, no dev feature) still ignores the flag. A Dev-feature export (the iOS-Dev, tvOS-Dev and Android-Dev presets) honours it. Honouring it is safe because the smoke switches ControlClient I/O off before it arms (prepare_device_boot_smoke), so a Dev export handed the flag cannot send a request. tests/test_device_boot_smoke.gd is unchanged; tests/test_device_boot_smoke_dev_export.gd pins the new behaviour, including the call site’s wiring. main.gd still carries the older comment “never in an export”.

Measured: before the change, a Dev pck under the release VM printed no marker at all (nothing the smoke could check). After it, DEVICE_BOOT_SMOKE armed=true captured=2 http_requests=0. And the store side holds: the real Store pck, handed --device-boot-smoke (release-smoke.sh --pck store.pck with no --store), printed no marker and reported MARKER-MISSING, so a store export stays inert under the release VM.

Proof: planted regressions

Each defect below was planted in a scratch copy of the tree (never in the repo, never committed), the debug VM was run the way gate.sh runs it, then scripts/release-smoke.sh was run on the same copy. Engine 4.7.1.stable, macOS 27.

Planted defect	Debug VM (gate.sh)	release-smoke.sh
none (the candidate)	green	PASS in 26 s: Dev marker seen, Store booted (exports 6.4 s and 6.7 s, runs 5.5 s and 5.4 s, pck 74.7 MB)
2026-10-09: _dev_flag_states calls debug_settings.graphics_report(...) without its != null guard	SCRIPT ERROR: Invalid call. Nonexistent function 'graphics_report' in base 'Nil' at main.gd:5677, exit 0, marker still printed. The gate’s SCRIPT ERROR grep fails it.	CRASHED: Dev pass exit 139 (SIGSEGV) after 1.0 s, no marker. The crash report is EXC_BAD_ACCESS, KERN_INVALID_ADDRESS at 0x20, the null-member read of the original incident.
A syntax error in main.gd	SCRIPT ERROR: Parse Error at boot, exit 0. The gate’s boot step fails on that line.	SCRIPT-ERROR in both passes: the release VM prints Parse Error and exits 0.
Exported-pck only: a boot line preloads a script under tools/, which every store and dev preset’s exclude_filter strips	green: no SCRIPT ERROR, marker printed, because the file exists in the source tree	SCRIPT-ERROR in both passes, exit 0: Parse Error: Preload file "res://tools/vfx_library/library_manifest.gd" does not exist. The game never booted.
Exported-pck only: a boot line load()s the same script by a runtime string	green, same reason	CRASHED in both passes, exit 139 after 1.1 s and 1.0 s. The log says Attempt to open script 'res://tools/vfx_library/library_manifest.gd' resulted in error 'File not found'.

Read this honestly. The 2026-10-09 plant is caught by both VMs: the debug VM prints the SCRIPT ERROR and today’s gate already fails on it, so for that exact regression the release smoke adds a second, harder signal (a real segfault, with the exit status) and nothing the gate would have missed. The class the gate cannot see is the last two rows: a defect that exists only in the exported pck (its file set is the source tree minus the export filter) or only in the release VM. That is what this smoke adds, in under a minute and before the build is uploaded. The two exported-pck rows are plants I chose to show that class; they are not past incidents.

Two measured details worth knowing:

• A scene ext_resource on a stripped script did not reproduce the failure: the pck loaded fine, so scene dependencies are packed whatever the filter says. A GDScript preload or a runtime-built path is not followed.
• A pck whose main script fails to parse exits 0. A smoke that looked only at the exit status would have passed the third and fourth rows, which is why the classifier also reads the log and the Store pass requires a boot file.

Related

• docs/reference/running-and-testing.md for the gate and the harnesses.
• docs/reference/builds-and-releases.md for the release process this should join.
• .claude/skills/bh-device-crash-repro for symbolicating a crash report (parse_ips.py) and reproducing a device crash by hand; its step 3 can now be scripts/release-smoke.sh.