Telemetry v2: no lost sessions, crash signal, run timeline, MetricKit, Sentry — design

Telemetry v2 — design

Status: approved direction (Chris, 2026-09-28/29). Detailed decisions below marked (Claude) were delegated (“make any further decisions yourself”) and are open to overrule. Findings that motivate this: docs/research/2026-09-28-telemetry-gaps-and-options.md.

Goal

Stop losing telemetry, and capture what we actually got stuck on: why ~half of sessions have no outcome, tester crashes, mid-run behaviour (XP bug, “invincible” run), real frame timing. Covers tvOS, iOS and web through our own Worker + D1; MetricKit (iOS) and Sentry (iOS) add native diagnostics where available.

Decisions (Chris)

1. Keep our Worker + D1 as the core (only thing covering tvOS). Unified client event queue + one POST /events batch route (option 1 of 3).
2. Random install ID on every event. Label becomes “Device ID, not linked to you”.
3. “Share diagnostics” ON by default, toggle in-game, privacy-notice line. No first-launch prompt.
4. MetricKit on iOS (Chris: may also inform tvOS via shared code paths).
5. Sentry included, free Developer plan (1 user; Toby doesn’t need access).

Decisions (Claude) — overrule freely

• Sentry: iOS only, not web. The web preset has extensions_support=false; turning it on enlarges the wasm and may need threads + COOP/COEP headers on the public demo, for a marketing-only build. tvOS isn’t supported by sentry-godot; the addon is excluded from the tvOS preset via exclude_filter. Ships dormant: no DSN in the repo = SDK not initialised. Chris creates the Sentry project and supplies the DSN.
• Old routes stay (/sample, /hitch, /gameplay, /encounter, /feedback) for builds ≤ current store build. New clients only use /events.
• Toggle lives in the pause menu (there is no player Settings screen; pause is reachable on every platform with focus navigation). Persisted as MetaState.share_diagnostics (default true) so it rides the existing tvOS NSUserDefaults save bridge.
• Turning diagnostics off: stop enqueuing, delete the queue file, stop sending install id on /config (config fetch itself continues — gameplay needs it), Sentry closed. Turning it back on generates a fresh install id (the old one is never reused).
• Dedupe via the events table, not unique indexes on legacy tables (see Worker).
• A quit to menu / background kill does not write a runs row (keeps balance data clean of partial runs); it writes a run_end event with a reason. runs rows stay “banked runs only”.
• dSYMs: ExportOptions.plist already has uploadSymbols=true. Work item D is verify whether the engine frames symbolicate (stock iOS templates, custom tvOS fork) and document; rebuilding export templates with symbols is out of scope tonight.

Architecture

reporters (telemetry.gd, gameplay_telemetry.gd, hitch_tracker via telemetry,
           dash_encounter_telemetry.gd, feedback, lifecycle, metrickit pickup, run timeline)
      │  EventQueue.push(type, data)
      ▼
net/event_queue.gd (autoload-like persistent Node; pure envelope/queue logic in RefCounted
                    net/event_queue_core.gd so it's GUT-testable headless)
      │  JSONL file  user://telemetry_queue.jsonl (LocalJson dir fallback on tvOS)
      │  flush: boot, every 10 s or ≥20 pending, on pause/background, after run end
      ▼
POST /events  {"v":1,"events":[…≤50]}   (text/plain, CORS-simple)
      ▼
Worker: for each event: INSERT OR IGNORE INTO events(id,…); if inserted and type is a legacy type
        → also INSERT into samples/runs/hitches/encounters/feedback with the new columns.
        responds {"ok":n,"dup":m} ; client drops acked ids from the file.

Envelope (every event)

field	meaning
id	16 random bytes hex, generated at push — dedupe key
t	type: sample, hitch, run, encounter, feedback, launch, run_start, run_end, menu, tutorial_step, shop_buy, background, foreground, prev_exit, metrickit
ts	unix ms at push
install	install id (MetaState)
launch	per-process random id (new every app launch)
session	run session (main._run_session) or "" outside a run
build, channel, dev	Sim_Const.BUILD, BuildChannel.current(), TelemetryDevice
auto	1 if the current run is autopilot/attract, else 0
d	type-specific payload (dictionary)

One session id everywhere: gameplay_telemetry and dash_encounter_telemetry stop minting their own _session and use _run_session.

Queue behaviour

• Push = append one JSON line (cheap; no rewrite). File capped at 1 MB; over cap, the oldest sample lines are dropped first, then oldest of any type.
• Flush sends up to 50 oldest lines; one request in flight; on 2xx with ok+dup == sent the file is rewritten without those ids. Failure: exponential backoff 10 s → 5 min, reset on success.
• Never blocks or errors gameplay: all failures silent (as today).
• Disabled (no push, no flush) in editor/headless/capture/replay/showcase, same gates as today, and when share_diagnostics is false.
• tvOS user:// is Caches (purgeable) → queue is best-effort there; acceptable.

Lifecycle & crash signal (replaces “no outcome”)

• launch event at boot: os version, model, first_launch, share_diagnostics, locale-free.
• Exit marker user://exit_marker.json, always armed while the app runs: {launch, session, run_time, state, recent (last fx kinds), backgrounded:bool, ts} — rewritten on run start, every sample (5 s), on background (backgrounded=true) and foreground (false). Removed on WM_CLOSE_REQUEST.
• Next boot, if the marker exists → prev_exit event: reason = killed_in_background when backgrounded (normal iOS/tvOS kill, app-switcher swipe) else crash (process died in the foreground: crash, watchdog, OOM). Carries the old launch/session so it joins to the dead run.
• run_start (mode, ruleset, auto, meta summary) in _new_run; run_end with reason ∈ death, story_complete, tutorial_cap, quit_menu, dev_restart, plus run_time, level. (killed/crash are inferred from prev_exit.)
• background/foreground events (with run_time if in a run), menu on start-menu show, tutorial_step when TutorialLessons advances (render-side poll of lesson index — no sim change), shop_buy (item id, cost, gold after) at the shop purchase point.
• The existing remote-config crash marker stays as-is (it drives blacklisting, a different job).

Run timeline (in the run event → runs.timeline_json)

• levelups: [[run_time, level, [offered ids], picked id], …] — logged render-side in _open_levelup / _on_upgrade_chosen (and the autopilot pick path).
• xp_by_type: new sim counter next to kills_by_type in _sweep_dead.
• damage_by_weapon: existing sim dict, now sent.
• bosses: [[type, spawn_t, kill_t|null], …] — new sim counter appended at the boss spawn functions and in _sweep_dead on boss death.
• New sim fields are plain counters outside the checksum, same class as kills_by_type; the determinism baseline must not move (no RNG draws, no ordering change).
• build_picks/weapons_list Worker truncation raised (400→4000, 120→1000).

Perf fields

• samples gains auto, gpu_ms, render_cpu_ms: RenderingServer.viewport_set_measure_render_time on the root viewport and every UPDATE_ALWAYS SubViewport (nebula backdrop, 3D ship) while the reporter is enabled; read viewport_get_measured_render_time_gpu/cpu summed over them, plus get_frame_setup_time_cpu (only on top of a real viewport measurement). 0 is sent as null. render_cpu_ms is the render-cost metric on our platforms. gpu_ms is always null on tvOS/iOS (the Metal driver has no timestamp queries) and on web (GLES only queries timestamps on desktop GL); it is kept because it fills for free on desktop Vulkan/D3D12/GL (corrected 2026-09-29, Task 6 review).
• proc_ms/render_ms keep being sent (continuity) but documented as whole-frame TIME_PROCESS; dashboard label fixed.
• Hitch sampling replaces “first 40 per run”: always send hitches that carry a new: first-use marker or pipelines > 0; otherwise the worst 2 per 30 s window; cap 200 per run.

MetricKit (iOS only)

• Native code via the existing hand-edited plugin-stub pattern (platform/tvos/patches/): a new patches/ios/dummy.mm for the iOS shell (build-ios/, currently stock dummy.cpp), with a __attribute__((constructor)) that registers an MXMetricManagerSubscriber. Payloads (MXMetricPayload, MXDiagnosticPayload) are written as JSONRepresentation files to <Caches>/metrickit/<kind>-<ts>.json, plus any pastPayloads/pastDiagnosticPayloads at registration.
• GDScript (net/metrickit_pickup.gd) at boot and on foreground: for each file, push a metrickit event (d.kind, d.json — truncated at 256 KB), delete the file once queued.
• bh-deploy / bh-appstore-release skill + patches/README.md updated so the iOS patch is re-applied after a full export (same as tvOS; includes the pbxproj .cpp→.mm fix).
• Guarded #if TARGET_OS_IOS so the file is safe if ever copied into the tvOS shell.

Sentry (iOS only, dormant until DSN)

• sentry-godot addon (version supporting Godot 4.7 + iOS, pinned) under addons/sentry/, symlinked into platform/tvos/addons/ like other shared dirs; excluded from the tvOS and web presets via exclude_filter.
• Initialised only if OS.get_name() == "iOS" and a DSN is configured (project setting / file not committed — Chris supplies) and share_diagnostics is true. send_default_pii=false, release = dark-cosmos@<BUILD>, environment = channel, tag install id.
• The implementer must verify the tvOS export still builds and boots headless with the addon present-but-excluded; if the GDExtension breaks the tvOS/web export or boot in any way, the Sentry item is parked (documented) rather than forced.

Worker + schema

• New table events(id TEXT PRIMARY KEY, ts INTEGER, type TEXT, install TEXT, launch TEXT, session TEXT, build INTEGER, channel TEXT, dev TEXT, auto INTEGER, data TEXT); data is NULL for legacy-mapped types (they live in their own tables), JSON for the rest. Indexes on (type, ts), install, session.
• Legacy tables gain install_id, launch_id, event_id (+ samples.auto, gpu_ms, render_cpu_ms; runs.timeline_json). ALTERs listed in the schema.sql migration block.
• POST /events: body ≤ 512 KB, ≤ 50 events, per-event validation with the existing coercion helpers; unknown types stored in events only. D1 batch() per request. Responds JSON counts.
• Feedback rate limit (5/10 min/session) preserved for feedback events.
• Deploy order (Chris OK required): D1 migration → verify PRAGMA → Worker deploy → smoke test /events with a session:"smoketest" event, read back, delete. Must happen before any build containing the new client reaches testers.

Privacy

• marketing/appstore/compliance.md updated: Device ID, Crash Data, Performance Data, Product Interaction, Other User Content (feedback text), all Not Linked, no tracking; third-party Sentry listed. Chris applies it in App Store Connect before the build ships.
• Privacy-notice text drafted in the spec follow-up for the site repo (~/Claude/bullet-heaven-site), not deployed tonight.

Testing

• GUT: event_queue_core (envelope, cap/drop order, ack rewrite, backoff, disabled gate), exit marker classification, hitch sampler, run timeline assembly, xp_by_type/boss counters through the real spawn→kill→_sweep_dead path, share_diagnostics toggle persistence + queue wipe + install-id rotation, determinism baseline unchanged.
• Worker node --test: /events insert, dedupe on resend, legacy fan-out columns, limits, unknown type, feedback rate limit.
• MetricKit: xcodebuild of the iOS shell with the patch (no signing) compiles; payload-file pickup tested in GUT with fixture files.
• scripts/gate.sh green before each merge; bh-reviewer per branch.

Out of scope

Retention/cohort dashboards (SQL views later), Analytics Engine migration, Sentry on web/tvOS, engine-template rebuild with debug symbols, deleting old Worker routes.