On-device perf scenario — design

On-device perf scenario — design

Date: 2026-10-01. Status: approved in brainstorm (Chris), pending spec review. Tracked in docs/RELEASE-1.0.1.md §1 (“On-device perf scenario in the Remote Playtest Console”).

Why

BUILD 235 telemetry shows a ~50-64 ms GPU stall per Supernova cast, worsened by webs and zones, but the evidence is pooled across uncontrolled runs. Remote-config tunables only apply at run start, the hitch log ignores stalls under 60 ms, and tools/perf_probe (which has the right A/B method) only runs on the Mac. We need the same interleaved, render-only A/B on the Apple TV (and iPhone/iPad), started from the Remote Playtest Console, with results we can trust.

Decisions (from the brainstorm)

• The schedule runs on the device, not driven step-by-step from the panel (relay latency and poll gaps would blur segment boundaries).
• Canned presets for the loadout, not the current run, so results compare across builds/devices.
• Results: live in the panel (device computes the table, publishes it via /status) and in D1 (one EventQueue event per segment). No new route between the control and telemetry Workers.
• v1 B-arms: Supernova shockwave, zone layer, web layer, and any v2 weapon look.
• Default 8 segments × 10 s, 5 s warmup, XP held at 0.

Architecture

Shared rig — render/perf_rig.gd (class_name PerfRig, static funcs taking main)

Extracted from tools/perf_probe/perf_probe.gd so the Mac probe and the device scenario run the SAME code:

• hold_targets(main, n) — n TYPE_CUSTOM dev targets (Sim.dev_spawn_custom, hp 1e9, speed 0, contact 0) re-pinned on a ring (TARGET_RING 70 px) around P1 every frame; respawns any lost.
• hold_webs(main, n, fade) / hold_zones(main, n) — keep sim.webs / sim.zones filled with n on-screen entries re-pinned around P1 (perf_probe’s current grid/fade/element logic, verbatim).
• hold_xp(main) — every pilot’s XP to 0.
• arm_ids() — the arm roster: nova_shockwave, zones, webs, plus every v2 look id (blade, scythe, nova, blaster, bolts, orbit).
• hide_arm(main, arm) — called from RenderingServer.frame_pre_draw: zeroes visible_instance_count on the arm’s MultiMeshInstance2D (zone_renderer._zone_layer, zone_renderer._web_layer, fx_layer.v2_look(w), bolt_renderer, orbit_look). nova_shockwave = the v2 nova look when fx_layer.v2_active(), else the v1 FxManager nova (needs a small FxManager accessor/hide seam if one doesn’t exist — the plan verifies). ZoneRenderer gains public accessors for its two layers rather than the rig reaching into _ fields.
• live_count(main, arm) — visible instances of the arm’s layer (validity check below).

perf_probe.gd is migrated onto PerfRig; its flags (--targets, --webs, --zones, --hide-v2, --ab) keep identical behaviour (smoke-run to confirm).

Presets — PerfScenario.PRESETS const

{name: {weapons: [wid…], evolve: bool}}, starting with:

• baseline — starting weapon only.
• supernova — nova, evolved.
• batch1_evolved — blaster, orbit, blade, scythe, nova, all evolved.

Applied via dev_reset_upgrades → dev_force_grant_weapon → dev_add_crystals (the real WeaponThresholds path, so evolutions apply exactly as a picked crystal card would). A unit test asserts every preset wid exists and every evolve: true preset actually ends evolved.

Runner — render/perf_scenario.gd (class_name PerfScenario extends Node)

Render-side, child of main, created on the first perf_scenario command. The pure parts live in PerfScenarioCore (RefCounted, no Node APIs) so they’re unit-testable:

• Scheduler: states SETUP → WARMUP → SEGMENT(i, side) → SETTLE → … → DONE | CANCELLED. Side order is ABBA repeated (A B B A A B B A for 8). Settle between segments = 0.8 s (longest look life, evolved Supernova 0.70 s, plus margin), budgeted in sim.run_time, not frames.
• Stats: per segment, from per-frame samples: p50/p95/max frame ms (main-thread _process delta), n_over_33, mean render_cpu_ms (existing Telemetry reader, net/telemetry.gd), gpu_proxy_ms = render_ms − render_cpu_ms, mean enemies/webs/zones, arm casts fired (from sim.render_fx_events), mean arm live instances, and HitchTracker.recent_tags() captured at the segment’s worst frame.
• Validity: a segment is invalid if sim advanced < 90% of wall time, main was frozen on a level-up/dimension panel at any point, or (B side, and A side for event looks) the arm’s look was never live. Invalid segments are re-run in place; give up after 2 × segments attempts → DONE with incomplete: true.
• Summary: per-side medians over valid segments and B − A deltas.

Command + status — main.apply_dev_command

• {cmd:"perf_scenario", preset, arm, segments=8, seg_s=10, warmup_s=5, targets=12, webs=0, zones=0} — validated (preset/arm in roster, numbers clamped: segments 2-20 even, seg_s 2-60, counts 0-400). Ignored unless BuildConfig.dev_tools(). A second start while running is refused (status note).
• {cmd:"perf_scenario_cancel"}.
• SETUP: dev_reset_upgrades, dev_clear_enemies, invuln on, spawns paused, preset applied. Prior values of invuln/pause are saved for teardown.
• build_caps() adds perf_presets (name → weapons) and perf_arms.
• build_status() adds perf_scenario: {state, scenario_id, preset, arm, seg, of, side, valid_count, rows:[…], summary:{A:{…}, B:{…}, delta:{…}}, incomplete, note} (null when never run).

Teardown

On DONE, cancel, run end, or return to menu: disconnect the frame_pre_draw hide, restore saved invuln/pause-spawn values, stop holding XP/targets/webs/zones and drop the held entries (dev_clear_enemies). Layer visibility needs no restore (hiding is per-frame).

Telemetry

Per segment: EventQueue.push("perf_seg", {scenario_id, preset, arm, seg, side, valid, frames, p50, p95, max, n_over_33, render_cpu_ms, gpu_proxy_ms, enemies, webs, zones, fired, live, worst_recent}). Verified: the telemetry Worker accepts any type matching /^[a-z_]{1,24}$/ and stores unknown types as JSON in events.data — no Worker, D1 or retention change. Build, session, channel and config_id ride on the standard envelope. No privacy-page/ASC-label change: same kind of data (device performance) already disclosed, and dev builds only.

Query later: SELECT data FROM events WHERE type='perf_seg' AND json_extract(data,'$.scenario_id')=?.

Panel — control/src/panel.html.js

New Perf tab, caps-driven: preset, arm, segments, seg length, warmup, targets/webs/zones, Run/Cancel; live progress strip; per-segment table + A-median / B-median / delta row; the scenario_id shown for D1 lookup. textContent only (existing XSS guard). Pane reveal uses display='block' (the known stylesheet gotcha).

Testing

• GUT: PerfScenarioCore scheduler (ABBA order, settle, re-run of invalid segments, give-up → incomplete), percentile/median maths, validity rules; preset table (wids exist, evolve presets end evolved via the real crystal path); command validation/clamping and dev-tools gate.
• GUT integration (headless): a 2 × 1 s scenario through main.apply_dev_command emits 2 valid perf_seg events (or explicitly-invalid ones where headless can’t render — the test asserts on the event shape and teardown, not frame numbers) and restores invuln/pause state.
• control/test/panel.test.mjs: Perf tab renders from caps; status table renders.
• perf_probe smoke run after the rig extraction (--targets=12 --hide-v2=nova --ab=1).
• Determinism baselines unaffected (nothing runs unless the command is sent); scripts/gate.sh green.
• On-device acceptance: one supernova × nova_shockwave run on the ATV dev app; 8 valid segments, a readable panel table, matching perf_seg rows in D1.

Out of scope (v1)

Cross-build history in the panel (ad-hoc SQL covers it); multi-arm (ABCD) runs — run once per arm; sim-side (non-render) arms such as REMNANT_FRAC, which would break the render-only guarantee.

Docs

docs/architecture/dev-tools.md § Remote Playtest Console gains a “Perf scenario” subsection; docs/RELEASE-1.0.1.md item ticked on merge.

Corrections (found while planning, 2026-10-01 — these override the text above)

• Dev commands key on kind, not cmd: {kind:"perf_scenario", …} / {kind:"perf_scenario_cancel"}.
• apply_dev_command had no dev-tools gate (the console’s arm path is gated upstream); the perf_scenario arm adds its own BuildConfig.dev_tools() check.
• main._process drains sim.render_fx_events before children run, so main forwards render_events to perf_scenario.note_events() next to telemetry.note_events().
• Telemetry’s render_ms is Performance.TIME_PROCESS, not a render measure, so gpu_proxy_ms is dropped: segments record raw frame_ms stats (≈ GPU time on a GPU-bound device), process_ms (TIME_PROCESS) and render_cpu_ms; derive splits offline.
• EventQueue.push is a no-op headless/in-editor, so the node takes an injectable push_event Callable (defaults to main.event_queue.push); tests spy on it.
• Presets are crystal counts per element (rules are independent, so the evolve rung fires on its own count): supernova = nova + fire 12 + light 18; batch1_evolved = blaster/orbit/blade/scythe/nova
  • fire 12, void 18, light 18, blood 18. The wallet is shared, so other owned weapons on those elements also cross their rungs — intended.
• v1 Supernova (v2 off, quality L3+) is hidden by setting pooled PlasmaNovaNode.visible = false (sticks until the node’s next reset); this also hides Scar Tissue novas. Fallback _spawn_ring (only when VisualLook.plasma_nova is off) is not covered.