AI‑First Operations and Infrastructure

Most studios have a pipeline that ends at “deploy” and a separate, human-shaped process that begins at “a player complained.” Here they are one track. A defect found by a machine at 04:00 and a defect reported by a person at noon enter the same queue, get the same fingerprint, and are ranked by the same rubric.

Everything on this track runs unattended except one station, and the automation is built to arrive there with the reasoning already done.

That station is meant to move. Today a person decides what a finding means and what to do about it. The direction of travel is that the machine also authors the fix and proves it against the replay seed that produced the failure — leaving exactly one human call at the end: ship it, or don’t. Not a person doing less thinking; a person doing the one piece of thinking that is expensive to get wrong.

The test driver is 8,743 lines living inside the shipping game. That is the whole trick, and it is worth being precise about why it matters.

A conventional mobile test rig — Appium, XCUITest, a Playwright device profile — stands outside the app and synthesises taps at screen coordinates. It cannot see why anything happened. It asserts on pixels, so it breaks when the art changes and passes when the art is right but the model is wrong.

Reveal Haven’s driver calls GameEngine.checkClick(x, y) directly, in scene coordinates — the exact method, with the exact signature, that a real player’s finger reaches. There is one other caller in the entire codebase: the canvas pointer handler. The driver is not simulating a player. It is entering the game through the same door.

A test that can’t be repeated is an anecdote. So the harness overrides every source of randomness it uses — which scene, which object, when to fumble, how long to dwell — with one counted stream drawn from a single seed, and it writes that seed into the report.

Re-running with a prior report’s seed replays its exact path: same scenes, same objects, same misses, same order. A crash stops being a story reconstructed from a log and becomes an integer you can replay on demand.

That is what makes a fix checkable. Land the change, re-run the seed that produced the failure, and the answer is unambiguous — it reproduces or it doesn’t. Without it, “I think that’s fixed” is the most anyone can honestly claim, and the next occurrence is the only way to find out.

Simulators do not get jetsam-killed. Thermal throttling, real memory pressure, real GPU drivers and the actual iOS memory manager only exist on actual hardware — which is exactly where a hidden-object game with 5504×3072 art plates falls over.

So the test lab is a single M4 Mac mini running three self-hosted GitHub Actions runners, with an iPhone 12 and an iPad Air 5 physically tethered to it. Two of the runners are labelled per device; the third does evaluation work.

The native build was being killed by iOS on an iPhone 12 at the first scene — a jetsam out-of-memory kill, peak resident memory 1,979 MB.

Instrumentation went in before any fix: a texture ledger reporting exactly what the GPU held at the moment of death. It reported 146 MB — the 5504×3072 backgrounds everyone suspected were a rounding error against a 1,979 MB process.

The real cause was the curtain-call transition: a blur effect wrapping the full-size content in nested effect nodes, rasterised at 3× device scale — roughly 600 MB of offscreen buffer allocated during a decorative flourish. Four measured builds followed, each verified by a full on-device campaign.

A game is a visual artifact, and “does this look right?” is the one question no assertion answers. The industry reflex is pixel-diffing — snapshot the screen, compare to a golden image, fail over a tolerance. On a game that inverts itself: every art change is a diff, every intentional improvement a failure, and the suite decays into golden files nobody trusts.

So the judgment has to be a model’s. What decides whether that is affordable is how much you make it look at. Record a session from launch and hand over the video, and you pay for thousands of frames to reach the one that mattered — every run, forever.

The engineering therefore goes into the cheap half of the problem: navigate to the exact state and capture that single frame. One image, not an hour of footage. The same question gets answered for a fraction of the tokens, and the cost of asking it doesn’t grow with the length of the session.

An evaluator is only as good as what it receives. “The app crashed” is not a diagnosis; it is a request for someone to go and reproduce it. So every surface — the nightly soak, a beta player’s phone, a person tapping Report an issue — returns a payload a model can actually reason over without asking a follow-up question, and does it while the always-on stream stays anonymous.

The naive version of “AI reads your telemetry” is a model call per event. That cost grows with your success, which is exactly backwards — the better the product does, the more you pay to watch it.

Here the model is the last stage, not the first. Two deterministic passes run ahead of it and absorb almost everything, and — the important property — every newly catalogued defect is absorbed by the cheap stage forever after. The cost of watching goes down as the catalogue matures.

A soak run and a beta tester report into the same stream, are fingerprinted by the same function, and are ranked by the same rubric. Below is the real feedback stream — a tester’s sentence and the harness’s own report, minutes apart, indistinguishable as queue items.

The fingerprint

Both kinds of report collapse to the same stable identity. An error message is reduced to its leading words, stopping at the first token that looks variable — an id, a number, a filename — so “preload FAILED name=sfx_object_found” and every other asset’s version of it become one defect: Audio:error:preload_FAILED.

That id is the spine of everything downstream. It is what a recurrence increments rather than re-reports, what a fix is filed against, and what tells the evaluator whether it is looking at something new — which is exactly what keeps the token cost in §08 tracking novelty instead of volume. 24 defects are catalogued this way.

Every completed soak run triggers an evaluation. A model reads the report, the crash log and the fingerprint registry, and emits a structured verdict — not prose. The driver validates it against a fixed enum and retries the model if it deviates: a classification, a confidence, a recommended action, a diagnosis, and evidence including the replay seed.

This is the shape the whole system is built toward: the machine does the reading, the human does the deciding. Not because deciding is hard to automate, but because it is the step where being wrong is expensive and unrecoverable.

And the deciding keeps getting narrower. The machine already reads the report and names the defect; the next step is that it also writes the fix and proves it against the replay seed from §03 — so what reaches a person isn’t a finding to investigate but a change with evidence attached, and the only question left is whether it ships.