Pipeline-compile tour: first-use compiles after boot

Pipeline-compile tour: first-use compiles after boot

A dev tool that boots the real game in a window on the Mac, lets the launch prewarm finish, plays a fixed walk (menu, run start, a hit, a level-up, pause, a boss, the death and its results card, back to the menu) and reports every step in which Godot still compiled a pipeline. A pipeline compiled after the launch cover lifts is a first-use stall on a device: the D1 hitch field pipelines carries the same sum (net/telemetry.gd adds all five monitors). PrewarmPass exists to pay those behind the cover; the tour finds the ones it misses.

It is one tool of the offline device-fidelity harness (docs/reference/fidelity-device-sim.md): it registers as pipeline-tour (tier window, scope global) and writes a fidelity-result/1 file.

Code: tools/shader_check/pipeline_tour.gd (the windowed driver), tools/shader_check/pipeline_tour_core.gd (pure: deltas, allowlist, verdict, table, result JSON; pinned headless by tests/test_pipeline_tour_core.gd), tools/shader_check/pipeline_tour_allow.json (known compiles), scripts/dev/pipeline-tour.sh (the wrapper), tools/device_sim/registry.d/pipeline-tour.json (the harness entry).

Run it

scripts/dev/pipeline-tour.sh                       # tv-menu, iphone-menu, tv-tutorial, then the verdict (about 2 min)
scripts/dev/pipeline-tour.sh --runs tv-menu        # one run
scripts/dev/pipeline-tour.sh --runs tv-menu --scan # also name the draw nodes behind every compile
scripts/dev/pipeline-tour.sh --report-only --out DIR   # merge the run files in DIR again, no window

Options: --out DIR (default $TMPDIR/pipeline-tour), --runs a,b, --allow FILE, --toggle LIST, --stop-after STEP, --scan, --hdr-boot 0|1, --timeout SEC (per windowed run, default 480), --no-lock (the device-sim harness already holds the window lock), --report-only.

Windowed Godot is serialised through scripts/dev/godot-window-lock.sh, kept in the foreground (App Nap throttles a background window), and project.godot / audio/bus_layout.tres (which Godot rewrites) are restored with git checkout -- after every run. It needs a window and a GPU renderer: headless reads zero and the report says so (inconclusive), it never passes.

Exit codes: 1 only when a NEW compile showed up; 0 otherwise, including an inconclusive tour (a run died, the prewarm pass never ran, the monitors read zero), which the report says in words; 2 for a usage error.

What it measures

Godot’s five Performance.PIPELINE_COMPILATIONS_* monitors: canvas, mesh, surface, draw, specialization. Every frame samples all five. A step owns the compiles of the frames between its start and the next step’s start (the next step opens three frames late, so the last draws of the one before land in their own row).

• Canvas, mesh, surface and draw gate the verdict. Each is a pipeline created on the render thread the first time a draw needed it: a stall.
• Specialization compiles on a background thread and does not stall a frame. It is printed and counted (metrics.specialization) and never gates.
• The sample lag: a compile counted at frame N happened while frame N-1 rendered. --scan therefore names the draw nodes that began drawing between the previous scan and this one (visible canvas items and layers, MultiMeshes with instances, emitting particles) for every frame a compile is counted in. They are the suspects, not proof: confirm by removing one and re-running.
• A boot row (under the launch cover) is shown for context and never judged. A tour whose boot compiled nothing is inconclusive: the monitors are dead (headless, or not a GPU renderer).

The steps

One windowed process is one run, because PrewarmPass pays once per process. A run is a (profile, route): profile tv (1920x1080, scale 1, TV title-safe insets) or iphone (2868x1320, scale 3, touch); route menu (a returning player on the start menu) or tutorial (a fresh save into the forced tutorial). The profile values are the ones tools/ui_capture/capture.gd uses (a test fails if they drift apart). The window is asked for the profile size; what the display gave is recorded as window= in every table.

Step	What the tour does
menu_idle	The start menu on show, then each row focused in turn.
start_survival	The Survival row pressed through the menu’s own path; the run’s first 1.5 s of sim time.
play	Ordinary fighting until 10 s of sim time, flown by a scripted human.
take_damage	Three swarmers on the ship so the sim lands a real contact hit.
levelup_open	The real panel, opened by a pending level (force_levelup).
levelup_close	A card chosen through the signal a card tap emits; play resumes.
pause	The glass pause menu opens (issue #98 tracks its first-open link on a device).
resume	The pause menu closes.
boss_intro	The Warden by dev command, invulnerable, for 5 s of sim time.
game_over_windup	The ship set to 1 hp with swarmers on it, until the death sequence BOOMs.
game_over_boom	The first 0.3 s after the BOOM (ripple, punch and flash first draw).
game_over_sequence	The rest of the explosion and the fade.
game_over_results	The results card on show (1.2 s).
return_to_menu	The results card’s own Menu press, back to the start menu.
tutorial_start, tutorial_play	The tutorial route: 4 s of the first lesson, then 6 s more (wall time).

The scripted human flies through the same touch seams InputRouter offers (_drive()), circling and aiming; it keeps the ship above 20 percent hp until the death step so the run lives long enough to reach the boss. The save is never read or written (MetaStore.disk_io_enabled = false is the first line of the tool), and a returning player comes through the one seam main.gd offers (_should_force_tutorial). PrewarmPass.force_available = true is set before main enters the tree: the editor binary would skip the pass otherwise.

Waits are budgeted in sim time (sim.run_time) while a run ticks and in wall seconds while it is frozen (menu, level-up, pause, results). The sim advances at a fixed 60 Hz however many frames an uncapped window draws, so a frame-count wait would measure the Mac’s frame rate, not the game. The tutorial route is the exception: it uses wall waits.

The fidelity lesson: boot the canvas HDR

The first version of this tour reported 13 compiles at the start of every run and a pile after it. They were an artefact. Canvas pipelines depend on the root canvas format:

• platform/tvos/project.godot (the project that builds iOS, tvOS, macOS and Android) boots with viewport/hdr_2d=true. The launch pass warms HDR pipelines and the run uses them.
• The root project boots 8-bit, and main.gd (_setup_backdrops) flips the root viewport to HDR at run start. Anything the launch pass linked for 8-bit links again at that flip: a ~77 ms frame at run start and a second wave of first-uses after it. That is what running the ROOT project directly on a desktop does (the editor, the dev harnesses); it is not what a device or the store macOS build does.

So --hdr-boot defaults to 1: the tour sets root.use_hdr_2d = true before main exists, which is the device boot. --hdr-boot=0 reproduces the root project’s own desktop boot (the table heading says boot canvas=8-bit (desktop boot)), useful only to see that relink. Never read an --hdr-boot=0 number as a device finding. (An early set of numbers in this work, 13 at start_survival and 3 to 6 at boss_intro, was exactly that; they are not device findings.)

The allowlist

tools/shader_check/pipeline_tour_allow.json, schema pipeline-tour-allow/1. It ships EMPTY: after the unlit-variant warm below, every step of every run reads 0, so any compile after boot is NEW and fails.

{"schema": "pipeline-tour-allow/1",
 "entries": [{"step": "levelup_open", "monitor": "canvas", "max": 1, "reason": "what compiles and why it is accepted", "issue": "#N"}],
 "runs": {"tv-menu": 1, "iphone-menu": 1, "tv-tutorial": 0}}

• An entry says: this step may compile up to max on this monitor in one run, and here is why. Anything above it, or not listed, is NEW and fails the tour.
• runs is the per-run cap: the most gated compiles a whole run may show. Per-step maxima alone add up to more than any run ever totals, so a regression that fits inside an unused allowance elsewhere would pass. A run over its cap fails with the step (run total). A run id not listed has a cap of 0, and an allowlist with any entry must list a cap for every default run (tv-menu, iphone-menu, tv-tutorial). Set a cap to the highest total a run of the sample showed.
• reason is required, issue is optional but should name the tracker issue. specialization is never allowed an entry, and a duplicate (step, monitor) is an error. A step name that is not a real step would silently never match, so test_every_shipped_allowlist_entry_names_a_listed_step fails on it.
• A step that ran but stayed below its max in every run is printed as STALE: that is run-to-run noise, or the compile was fixed, in which case tighten the entry or delete it.
• A broken allowlist makes the report inconclusive, never a pass. A WARN headline reads N known compile(s) after boot, within the allowlist (run total/cap: ...): it never says the compiles are absent.

Run-to-run noise is real, and it is in WHERE a compile lands, not in how many there are: the same first-uses land in different steps depending on what the fight happens to reach first. That is why the per-run total and its cap, not the per-step maxima, are the regression signal.

The result

{out}/result.json, fidelity-result/1, profile global (one row, however many runs).

Status	Meaning
fail	At least one NEW (un-allowlisted) gated compile after boot. The only status that exits 1.
inconclusive	No run, a run that did not finish, an unreadable allowlist, or a boot that compiled nothing.
warn	Only allowlisted compiles remain (within each step’s max and each run’s cap): known, still unfixed.
pass	Nothing gated compiled after boot.

metrics carries runs, steps_checked, boot_gated, gated_after_boot, violations, known, stale_allow, specialization, by_step (<run>.<step> to a count) and by_run (the gated total after boot per run, clean runs listed as 0), run_caps-judged totals in the violations, and tutorial_lesson_reached (the lesson a tutorial run got to). caveat is always the limits section below in one line. The raw per-run files (run_<id>.json, schema pipeline-tour-run/1) hold the boot row, every step with its deltas, frame count and slowest frame, the logged compile frames (with the death sequence’s state, and the --scan suspects), the engine and renderer, and the problems met.

Finding the cause of a compile

This is the loop that found the first fix (the death ripple, below). Each step is one run of one profile:

1. --scan (add --runs tv-menu for speed). The run file’s frames[] holds, for each frame a compile was counted in, the step, the frame time, the sim time, the death sequence’s state, and the nodes that began drawing (new).
2. Attribute before editing. --toggle removes one candidate cause and the per-frame log of the toggled run says whether it was the one: comfort_reduce (no ripple, punch or flash in the death), no_shockwave, no_punch, death_off (straight to the results card), no_boss, kill_direct (hp set to 0, no contact hit), hp_low, ring_off, ring_first, ripple_early (the run’s first ripple triggered in ordinary play, tiny) and early_death (die with almost nothing else drawn yet) and the inputs of the run’s own DeathSequence: ds_no_layers, ds_no_targets, ds_no_kit (hides nothing, so the player aura light stays on), ds_no_ships. Toggles script hp and nodes: they are attribution aids, not a route a player takes.
3. --stop-after STEP ends a run after a step, to iterate on the early rows without the whole walk.
4. Confirm a fix with the full default tour, not only the step it targets: a compile moves to the next first-use, and the totals can shift by one or two without anything being wrong.

What it found

All numbers: Mac, Apple M5, Metal, Godot 4.7.1 editor binary, --rendering-method mobile, HDR boot.

Fixed 1: the first death ripple linked two canvas pipelines at the game-over frame. Family 15 of PrewarmPass draws the annulus mesh as a Node2D behind its own tiny BackBufferCopy. The run draws it as ShockwaveLayer, a CanvasLayer with a full-screen ColorRect on the frame after trigger() and the annulus behind a rect BackBufferCopy on the frame after that. Neither matched. Family 15a in render/prewarm_pass.gd builds two real ShockwaveLayers with the run’s own constructor, one left in the first state and one advanced to the second, at a strength of 0.01 (strength 0 submits no annulus). Pinned by tests/test_prewarm_shockwave_layer.gd. Default tour, game_over_windup and game_over_boom: 1 and 1 before, 0 and 0 after, on both profiles.

Fixed 2 (review round 1): a first death with no boss alive linked 3 to 6 canvas pipelines in the one frame of the BOOM. Before this fix, --toggle no_boss (death at about 18.6 s of sim time, after 10 s of play, a level-up and a pause, no Warden) read:

--toggle no_boss, before	tv-menu	iphone-menu
game_over_windup (the BOOM frame, booked a step early before the ordering fix)	4 NEW	6 NEW
everything else in the death	0	0

The default walk spawns the Warden before it dies, which hid this. Attribution (tv, no_boss, each switch alone): comfort_reduce (no flash, ripple or punch), no_shockwave and no_punch all still read 4, and all three together still 4; death_off read 0. So it was the sequence itself, and the --scan of the frame named no node that began drawing. Warm instances of the sequence in every drawn state (overlay, full_fx, ship, targets, crystals) did not change it. Dropping one input of the run’s sequence did: ds_no_layers 4, ds_no_targets 4, ds_no_ships 5, ds_no_kit 0. The kit holds the player’s aura light (HeroLights.player_aura()), which the BOOM hides. With no boss alive that is the LAST light in the canvas, and every canvas item then switches from its lit pipeline variant to the unlit one. Those unlit variants were never linked anywhere (4 on tv, 6 on iphone: the count depends on the profile’s drawn items). A boss intro keeps a boss light on, which is why a death after the Warden never saw it.

The fix, in render/prewarm_pass.gd: the pass hides the aura light for its last UNLIT_FRAMES (8) frames, so the lit variants link first and the unlit ones link under the launch cover. Family 21b also now holds the death sequence the way the run builds it (overlay, full_fx, a dummy ship, targets and crystals) in the wind-up, the BOOM frame and the explosion, instead of one instance advanced 0.5 then 0.4 s in _ready (which drew only its last state). Pinned by tests/test_prewarm_unlit_death.gd.

HDR boot, after the fix	tv-menu	iphone-menu
--toggle no_boss, every game-over row	0	0
--toggle early_death, every game-over row	0	0
default tour, every step including start_survival, take_damage, levelup_*, resume, boss_intro	0	0
tv-tutorial (boot and first lesson only)	0
boot canvas pipelines (under the cover, not judged)	154 (was 107)	154 (was 107)

The unlit warm also removed every compile the old allowlist carried for issue #112 (start_survival 2, take_damage 1 to 3, levelup_open 1, levelup_close 1 to 2, resume 0 to 1, boss_intro 0 to 3): they were the same unlit variants first linking when the aura or a boss light changed. That is the Mac result; whether the iPhone first-kill and dmgnum bursts in D1 go too is a device question (below).

The price: 47 more canvas pipelines linked under the launch cover (107 to 154 on the Mac). On a device that is extra boot time behind the cover; the Mac cannot size it, because a Mac compiles faster than an iPhone or an Apple TV and the tour does not time the boot. Measure launch time on a fresh install before shipping (device section below). PrewarmPass.UNLIT_FRAMES is a const on purpose: it is a structural warm window, not a feel value; setting it to 0 turns the unlit warm off.

Not reproduced on the Mac: the pause glass (issue #98). The device-only bake link reads 0 here.

Limits (what this tour cannot tell you)

• It runs the Mac EDITOR binary on Metal. That binary also warms the dev-only prewarm families (25, every web look, webs-lite), so a clean tour is optimistic. Counts match a store device in kind, not magnitude. A device can still compile what this run does not.
• The window is the profile’s pixel size, not its hardware: no 120 Hz display, no device memory pressure, no tvOS fork or iOS shader baker (issue #98 is a baker-only bug).
• The run is flown by a scripted human through the touch seams: no real touch controls, no pad prompts, no Siri Remote focus. Screens the walk never opens (codex, shop, settings, other menus, the other weapons’ first fire, other bosses, other areas and Dimensions) are not covered.
• The tutorial route has no scripted input. It boots into the first lesson and stays there (the run file’s tutorial field and metrics.tutorial_lesson_reached say lesson 0), so its 0 says nothing about later lessons or the tutorial’s objective marker.
• It runs the BUNDLED content: in the editor RemoteConfig uses a null store and never fetches, so tunables a live remote-config patch changes on store devices are not what the tour draws.
• Some steps use scripted state (force_levelup, the Warden by dev command, hp set low), and the menu route skips the first-encounter codex cards.
• SPECIALIZATION is reported, never gated.
• Where a compile lands is noisy by one to three; the per-run total is the stable number.
• The device table in pipeline_tour_core.gd is a hand copy of tools/ui_capture/capture.gd, pinned by a test against it; tools/device_profiles/profiles.json is the better single source once it is adopted.

The device side: the pl: hitch tag

The D1 hitch field pipelines is the sum of all five monitors, so a hitch with pipelines > 0 could be a compile that stalled the render thread or only the background SPECIALIZATION kind. net/telemetry.gd now also notes a short tag in the hitch’s recent string, beside the existing pipelines tag: pl: plus one letter and a count for every monitor that moved on that frame, always in the order below.

Letter	Monitor
c	canvas
m	mesh
s	surface
d	draw
b	background (specialization)

pl:c1d2 is one canvas and two draw compiles; pl:b3 is three background compiles and nothing that stalls on a compile; a count of 9 means nine or more. The longest form, pl:c9m9s9d9b9, is 13 characters. It splits counts the hitch already carries, so there is no schema change (recent is the existing free-text field, cut at 400 characters by the Worker) and no new kind of data.

The tag goes through HitchTracker.note_plain, so it is never marked new: (a new: mark makes the sampler send any stall in the next second, which a per-combination first sighting would trigger for unrelated stalls). The existing pipelines tag keeps its first-sighting mark. A run of consecutive compile frames takes two ring entries per frame instead of one, because the two tags alternate and the tracker only drops a tag that repeats the previous one. The ring holds 24 and keeps first sightings.

The tour cannot see telemetry. tests/test_telemetry_pipeline_tag.gd drives Telemetry._track_frame with a fed reading per frame (the monitors read zero headless) and pins the format, the cap and the ordering.

What only a device can verify

1. Fresh install of a Dev build (skill bh-fresh-install-stall-check), then 5 fresh-process deaths each on iPhone and Apple TV, a mix of deaths before and after a Warden: no game_over hitch row with pipelines > 0 (D1 had 17 rows of 63 to 616 ms, median 93 ms).
2. Launch time. The pass now links about 47 more canvas pipelines under the cover: compare time to the start menu on a fresh install (Apple TV and the A12/A17 phones) against the previous build. If it is a problem, PrewarmPass.UNLIT_FRAMES = 0 is the switch.
3. Whether the first-hit, first-kill, first level-up and first wave banner hitches in D1 (the old issue #112 list) are gone, as the Mac says.
4. Issue #98 (the pause glass) stays open: the tour reads 0 on the Mac, so it can neither confirm nor close it.
5. That pl: tags reach D1 on a hitch, which needs a build carrying the tag.

Adding a step or an allowlist entry

A new step needs a name in PipelineTourCore.STEP_ORDER, a _next("name") in the route in pipeline_tour.gd, and a row above. An allowlist entry needs its step to exist. The tool’s own tests (tests/test_pipeline_tour_core.gd) fail on a step an entry names that is not listed, and on a shipped allowlist entry without a reason.