Skip to content
Dark Cosmos — Dev Docs

Dev Tooling vs AAA Practice — Gap Audit

Dev tooling vs AAA practice (2026-10-15)

Section titled “Dev tooling vs AAA practice (2026-10-15)”

What Dark Cosmos uses to build itself, set against what large studios use to stay polished, efficient and organised. Follows 2026-10-07-dev-infrastructure-meta-audit.md (multiplayer/class readiness); this one is the broader “studio toolchain” comparison.

Tags: [Certain] verified in repo · [Likely] industry practice from general knowledge, not re-researched this session · [Guessing] judgement.

Headline

Section titled “Headline”
  1. The tooling is not the weak point; closure is. [Certain] Since 2026-09-15: 1,925 commits, 337 (17.5%) start with fix/revert, 155 git worktrees (75 on 10-07), NEXT_TASKS.md 53 KB. None of the 10-07 audit’s Tier 0/1 tooling items shipped (no prune-worktrees.sh, net-pair.sh, bots.sh, pre-push gate stamp). AAA studios solve this with production discipline (one tracker, triage, definition-of-done, content lock), not more tools.
  2. Biggest real gaps vs AAA: crash reporting, a single bug tracker, automated perf budgets, localisation, accessibility. Everything else is at or above what a studio this size would have.

Inventory — what we have

Section titled “Inventory — what we have”
Area Ours AAA equivalent Verdict
Engine/arch Godot 4.7/4.8, custom tvOS fork, pure deterministic /sim, seeded RNG streams Unreal/in-house, deterministic sims mostly in RTS/fighting games Ahead for a solo game (replay, bots, lockstep all fall out of it)
Automated tests GUT, ~580 scripts, gate.sh, test-count guard, determinism pins Unit + functional tests, Gauntlet-style automation On par
Bots / soak soak, SkilledAutoPlayer, learning autopilot (MAP-Elites personas), story_auto, difficulty_probe, economy_sim Automated playthrough bots, balance sims (e.g. EA/King ML playtesting) Ahead of most mid-size studios
Live tuning Remote Playtest Console + MCP, registered tunables, remote-config patches, live SFX audition Console variables, hot reload, live-ops config (PlayFab, in-house) On par
Telemetry Worker + D1: samples, hitches, deaths, run timeline, MetricKit Game analytics + perf telemetry dashboards On par; dashboards are query-only (bh-telemetry)
Content/data bible.json + design-bible tool, tunable_docs, Sound Map Data tables, design spreadsheets → importers On par; no schema validation in the gate (10-07 #12, not done)
Visual QA ui_sweep, ui_capture, bh-visual-qa, device-profile fit tests, vfx_library roster Screenshot regression, art review boards On par
Asset tools glow_bake, ship_preview, bg_preview, compare harnesses, ElevenLabs/synth SFX pipeline DCC tools (Houdini, Substance), asset validators Fine for 2D scope
Code review bh-implementer → bh-reviewer in worktrees, adversarial review loops Mandatory peer review (Swarm/Gerrit) On par
Release scripts/release/*, bump-build.sh, ASC tooling, TestFlight, Android tester APK Build farm + release managers Works; manual, single-machine
Docs Starlight site, architecture docs, research docs, CLAUDE.md, skills Confluence/wiki Above average

Gaps — what AAA does that we don’t

Section titled “Gaps — what AAA does that we don’t”

Ranked by value for this game right now.

1. Crash reporting with symbolication and per-build crash rate — HIGH [Certain gap]

Section titled “1. Crash reporting with symbolication and per-build crash rate — HIGH [Certain gap]”
  • Ours: Sentry wired but dormant (SentryBoot, iOS-only, consent-gated), MetricKit arrives a day late, hand-symbolicated. The 10-14 first-play study could not explain the iPad Air death: “telemetry can’t say more”.
  • AAA: Backtrace / Sentry / Crashlytics on every platform, auto-symbolicated, crash-free-session rate per build as a release gate. [Likely]
  • Do: turn Sentry on for Dev + TestFlight first (consent rules unchanged), upload dSYMs in scripts/release/, add “crash-free sessions ≥ X% on the candidate” to the release pre-flight. Privacy page + ASC label must be updated if store players are included.

2. One bug/task tracker — HIGH [Certain gap]

Section titled “2. One bug/task tracker — HIGH [Certain gap]”
  • Ours: work is spread across RELEASE-1.1.md, 53 KB NEXT_TASKS.md, ~30 📍 memory threads, research docs, and 155 worktrees. GitHub Issues unused (0 issues).
  • AAA: Jira/Hansoft/Shotgrid — every bug has severity, priority, owner, build found/fixed, and goes through a triage meeting; milestone burn-down. [Likely]
  • Do: one tracker (GitHub Issues + a Project board is free and gh-scriptable, so agents can file and close). Markdown queues become views, not stores. “Waiting on Chris” becomes a label, which also covers 10-07 #4.

3. Automated performance budgets — HIGH [Certain gap]

Section titled “3. Automated performance budgets — HIGH [Certain gap]”
  • Ours: sim_bench, perf_probe, on-device A/B via the console, hitch telemetry. All run by hand; gate.sh has no perf budget.
  • AAA: per-platform frame budgets (CPU/GPU/memory) with nightly automated captures on target hardware and per-build graphs; a regression fails the build. [Likely]
  • Do: (a) sim_bench worst-case scenario in the gate with a tick_ms ceiling (headless, cheap); (b) a per-BUILD p95 frame time / hitch-rate chart from existing D1 samples + hitches tables, checked in release pre-flight. On-device nightly runs come later (needs a device farm).

4. Localisation — MEDIUM, cost grows every release [Certain gap]

Section titled “4. Localisation — MEDIUM, cost grows every release [Certain gap]”
  • Ours: 0 tr() calls, no string tables. Everything is English literals.
  • AAA: loc-ready from day one: string IDs, pseudo-localisation to catch truncation, loc vendor pipeline. [Likely] Retrofitting is the expensive path; every new UI string adds to it.
  • Do (post-1.1): Godot CSV/PO translations, a lint that flags new literal strings in ui/ (ratchet like the UI-tokens literal ratchet), pseudo-loc pass at the device fit-test profiles.

5. Accessibility — MEDIUM [Certain gap]

Section titled “5. Accessibility — MEDIUM [Certain gap]”
  • Ours: “Reduce shake & flashes” (render/comfort_fx.gd) only.
  • AAA: Xbox Accessibility Guidelines / Game Accessibility Guidelines checklist: colour-blind modes, text scale, remapping, hold-vs-toggle, subtitles for audio cues. [Likely] Console cert and storefront accessibility labels increasingly ask for these.
  • Do: a colour-blind pass matters most here — the game reads danger/element by neon hue. Then text scale. Track against the GAG checklist as a doc.

6. CI / build farm — LOW now, HIGH before consoles [Certain gap; deferral still valid]

Section titled “6. CI / build farm — LOW now, HIGH before consoles [Certain gap; deferral still valid]”
  • 10-07 deferred CI (solo committer; Linux runners can’t test ARM determinism). Still right. But the cheap substitute (pre-push gate stamp, 10-07 #10) never shipped. Ship that.
  • Revisit when console ports start: every target needs a nightly build, or platform breakage is found at release time.

7. Certification checklists — LATER

Section titled “7. Certification checklists — LATER”
  • Console TRC/TCR/Lotcheck when PS/Xbox/Switch happen. Apple review rules are already tracked (guideline 2.3.1 note in CLAUDE.md). Nothing to do yet.

8. External playtesting at scale — MEDIUM [Likely gap]

Section titled “8. External playtesting at scale — MEDIUM [Likely gap]”
  • Ours: scripted playtest doc, Toby, in-game feedback, bh-feedback-triage, replays. ~18 human runs of calibration data.
  • AAA: user-research labs, recruited cohorts, structured surveys, heatmaps. [Likely]
  • Do: a recruited TestFlight cohort with a short post-run survey and a death-position heatmap built from the existing run timeline. Cheap, and the autopilot gap study needs more human data anyway.

Not worth copying

Section titled “Not worth copying”
  • Perforce / Git LFS: .git is 285 MB, 2D assets. Git is fine. [Certain]
  • Large asset validation pipelines, DCC integration: scope doesn’t need them.
  • Producer layer / meetings: the equivalent here is a weekly 15-minute triage of the tracker (gap 2), done by Chris with Claude preparing it.
Section titled “Recommended order”
  1. Pre-push gate stamp + worktree prune script (both S, both from 10-07, unshipped).
  2. GitHub Issues as the tracker; migrate NEXT_TASKS.md + 📍 threads in one pass.
  3. Sentry on for Dev/TestFlight + dSYM upload.
  4. sim_bench budget in the gate + per-BUILD perf chart.
  5. Post-1.1: localisation scaffolding + literal-string ratchet; colour-blind mode.