The Agent Bridge: Playing a Game You Can't See

The obvious way to have an AI agent playtest a game is to show it the game. Screenshot the canvas, hand the image to a vision model, ask what it sees, decide a move, press a key, screenshot again. It works, in the sense that a demo works.

For a reflex-heavy survivors game it is also close to the worst possible design. Every decision costs an image. Context accumulates until the session degrades. The model is reading a rendered frame to recover state the engine already knows exactly — enemy positions, hitpoints, cooldowns — and recovering it worse. And the moment anything interesting happens, it has already happened, because between the screenshot and the keypress the sim ran on.

CandyRush is a deterministic, fixed-timestep simulation. That fact makes a different design available: pause the sim, read the state as JSON, inject input through the same buffer the replay system uses, and advance an exact number of ticks. Structured text in, deterministic ticks out. No pixels anywhere in the loop.

That surface is the agent bridge, and it is live in production right now.

JS entry points

window.__candy*

Cold menu → live run

2.0s

zero clicks

Step 600 ticks

150–310ms

was 21,500ms

EditMode tests

178

10 bridge test files

Browser vet rounds

25 fixes, F1–F16 + R3F1–R3F9

Errors it attributed

277

per scene load, GH #610

The shape of it

The loop is four steps, and none of them involve looking at anything:

// 1. read — the latest snapshot, already parsed
const s = window.__candyState;

// 2. decide — plain arithmetic over structured data
const threat = s.enemies.filter(e => e.damageReadyInTicks < 10);
const dir = fleeFrom(threat, s.player);

// 3. inject — a deterministic input for the tick the sim consumes next
window.__candyInput({ moveX: dir.x, moveY: dir.y });

// 4. step — advance exactly N ticks, then emit a fresh snapshot
window.__candyStep(6);
await window.__candyWait();

The whole surface is compiled out of production builds behind #if UNITY_EDITOR || DEVELOPMENT_BUILD. In a dev WebGL build it is additionally gated on the page URL, which is what lets one deploy serve two audiences.

Driving it from a browser today

The dev channel is deployed and public. The plain URL plays the game normally. Adding one query parameter arms the bridge:

URL	Bridge
`unity.irsik.software/games/candyrushdev/`	off — plays normally
`unity.irsik.software/games/candyrushdev/?agent=1`	on
Editor, desktop players	on, always

?agent=1, ?agent=true, a bare ?agent, or an agent-mode whole path segment all arm it; ?agent=0 is an explicit opt-out that beats the path form.

That gate exists for a mundane reason. Before it, every human who loaded the dev channel to play was paying for a full state snapshot — serialized, then console.log'd as a CANDY_STATE line — every six ticks, about ten times a second, producing output no person would ever read.

It is worth saying plainly: this is not a security boundary. The bridge is already absent from production builds, and anyone who can load the dev channel can append the query themselves. It stops a player getting agent behaviour by accident. That is all it is for.

A session, from a cold page load

I ran this against the live deploy while writing this post. Every value below is what came back.

// Pin the world before it exists. Survives a page reload (PlayerPrefs → IndexedDB).
window.__candyPinSeed(777001);
await window.__candyWait();
// → {"ok":true,"cmd":"pinSeed","seed":777001,"seq":1}

// Boot a run from the cold main menu. No lobby, no clicks, no Play button.
window.__candyStartRun({ characterId: 0 });
await window.__candyWait();
// → {"ok":true,"cmd":"startRun","characterId":0,"arcId":"","status":"loading","seq":2}

// Poll until the scene lands.
while ((window.__candyState || {}).runState === "Menu") {
  await new Promise(r => setTimeout(r, 250));
}

Two seconds later, from a cold menu:

{ "runState": "Paused", "tick": 0, "seed": 777001, "pauseOwners": ["agent"] }

Read that last line carefully, because it is the whole design in one object. The run is frozen at tick zero, held by a pause token that belongs to the agent and nobody else. Zero uncontrolled real-time ticks have run. The seed is the one that was asked for. Nothing has happened yet that the agent did not cause.

That property was not free — it is fix R3F1, and the section on browser vets below explains what it cost to get.

The API surface

Twenty entry points, all installed on window. Every one is a queued command, not a synchronous call.

Reading state

telemetry

Call	Effect
`__candyState`	A property, not a function. The latest snapshot, already parsed.
`__candyDump()`	Requests a fresh snapshot and synchronously returns the previous one.
`__candySetMaxEntities(n)`	Raise or lower the nearest-N cap on per-entity lists. Default 15.
`__candyVfxAudit()`	Names every `VisualEffect` in the loaded scene. See the bug hunt below.

Acting

control

Call	Effect
`__candyInput(...)`	Injects movement for the tick the sim consumes next. Movement is the only per-tick input — CandyRush is an auto-attack survivor, so there is no fire, dash, or ability button.
`__candyLevelUp(action, index)`	Resolves the level-up picker: Pick, Skip, Reroll, Banish. Reports whether it actually applied, so a charge-gated no-op is never mistaken for success.
`__candyStep(n)`	Advances exactly n ticks by driving `SimulateOneTick()` directly.
`__candyStepUntil(...)`	Steps up to a budget, stopping the moment a condition trips, and reports which.
`__candyPause()` / `__candyResume()`	Pause via a dedicated owner token, so it composes with in-game pauses instead of clobbering them.
`__candyStartRun(...)`	Boots a run headlessly from any phase, cold main menu included.
`__candyCheat(name, ...)`	Nine sim-safe cheats, applied at the top of the next simulated tick.
`__candyWait(timeoutMs)`	A promise. The supported way to await a queued command.

Time travel

reconstruction

Call	Effect
`__candyPinSeed(seed)`	Pins the content seed for the next run boot. Survives a page reload.
`__candyExportReplay()`	Exports the run: seed, character, arc, and the input, level-up, and cheat streams.
`__candyRewind(k)`	Deterministically restores the sim to `currentTick - k`.
`__candyLoadReplay(json, ...)`	Loads an export and either watches it at wall-clock speed or fast-forwards to verify it.

Every emit also writes one CANDY_STATE <json> console line, so an agent that can only read a console — with no access to window at all — is still fully supplied.

What the snapshot actually carries

This is the part that decides whether the whole idea works, and the temptation is to dump engine internals and let the agent sort it out. The bridge deliberately does the opposite.

{
  "tick": 12456,
  "runState": "InRun",
  "pauseOwners": [],
  "seed": 3921184017,
  "player": { "x": 12.3, "y": -4.1, "hp": 84, "maxHp": 100, "level": 7,
              "killsThisRun": 41, "damageDealtLastTick": 12.5,
              "abilities": [ { "name": "WoodenWand", "cooldownRemainingTicks": 34, "ready": false } ] },
  "enemiesTotal": 137,
  "enemiesInContact": 2,
  "incomingContactDps": 7.5,
  "enemies": [ { "id": 91, "x": 15.0, "y": -2.0, "type": "gumdrop", "hp": 12,
                 "vx": -0.05, "vy": 0.02, "contactDamage": 5,
                 "damageReadyInTicks": 34, "marked": false, "behavior": "chaser" } ],
  "run": { "segmentIndex": 3, "archetype": "Hunt", "quotaProgress": 2, "quotaTarget": 5,
           "gates": [ { "id": 0, "x": 12.5, "y": -30.0, "lockState": "Locked", "leadsTo": "Boss" } ] }
}

It is a contract, not a dump. Every value is derived and agent-ready rather than mirroring an engine field, because raw fields force the agent to re-derive using constants it cannot see — a hidden coupling that breaks silently the moment the game is rebalanced.

A few of the choices are worth pulling out, because each one is a bug that would otherwise be waiting:

enemies[].vx/vy is movement realized on the last tick, after flow-field steering, separation, and obstacle pushout — not the enemy's intent. A chaser re-aims at the player every tick, so extrapolating raw intent extrapolates wrong.
enemiesTotal and incomingContactDps are whole-swarm, while the per-enemy list is a nearest-15 sample. Computing incoming damage from the sample while hundreds are in the field reads as safe right up until you die.
run, boss, and result are omitted, never zeroed, when they do not apply. A zeroed run reads as "segment 0 of a live run"; a zeroed result reads as "0 stars".
pauseOwners names who holds the sim — ["agent"] versus ["agent","levelup"]. That is the difference between "call resume" and "call levelUp", and a bare runState:"Paused" cannot express it.
All rates are per-tick, so they compose with __candyStep(n).
NaN and ±Infinity publish as 0, plus a nonFinite key listing the offending paths. Raw NaN is not legal JSON and used to cost the agent the entire snapshot. Publishing null was rejected for a sharper reason: a comparison like hp < 20 reads true against null, turning a broken sim into a confident wrong decision.

Determinism is the actual product

Any harness can poke a running game. What makes this one a test instrument is that a session is reproducible, and that took a specific set of decisions:

Input is keyed to the tick the sim will read next, mirroring how hardware sampling keys its own frames. An agent-injected frame is consumed exactly like a sampled or replayed one.
Injection suppresses hardware sampling, so a stray keypress on the page cannot contend. The agent's input is authoritative and recorded, so the session replays identically.
Stepping never touches wall-clock time. It takes the agent pause token first, so the frame-driven loop cannot also tick between steps.
Cheats apply at the top of the next simulated tick, never from the command callback. Spawning and killing consume the deterministic RNG; a cheat applied at frame time would advance the RNG off-tick and silently diverge every replay. Each application is recorded with its tick and shipped in the export.

The proof of it is unglamorous and exactly the right shape: two runs pinned to seed 777001, stepped with zero input to the same tick 1800, compared byte-for-byte. All 25 enemy positions matched exactly. Zero diffs.

Stopping on something interesting

The naive loop — step six ticks, read, decide, repeat — costs a full round trip per decision. __candyStepUntil collapses that into one call:

__candyStepUntil({ maxTicks: 600, stopOn: ["hpBelow:40", "levelReached:5"] });
const { result } = await __candyWait();
// { ok:true, cmd:"stepUntil", requested:600, stepped:214, tick:8931, stopped:"hpBelow" }

stopOn is a closed set, not a predicate language: hpBelow:N, levelReached:N, segmentChanged, gateUnlocked, bossSpawned, quotaMet, plus gameover and levelup which always halt. Exactly one reason wins, by a fixed precedence — dying also drops HP below any threshold, and precedence is what stops that reporting the useless hpBelow.

Position predicates were deliberately left out. An agent can compute those client-side between calls, and every predicate added to the closed set is a predicate the sim has to evaluate every tick forever.

The bug that proves the point

In a dev WebGL build, Unity's native VFX runtime printed this on every Game-scene load:

Invalid VFX Particle System. It is skipped.

277 times, in a single burst, naming nothing. No GameObject, no path, no asset, no GUID. The string comes from Unity's own C++ module — it lives inside the shipped .wasm and cannot be reworded from project code. It flooded the on-screen Development Console until real errors were invisible, which was the entire cost: nothing looked wrong in the game.

The leading theory was that a portal display mode was failing on WebGL. An agent driving the bridge produced this table instead:

Observation	Measurement
At the main menu, before any field loads	0 errors
On Game-scene load	277 errors, one ~33 ms burst, all at tick 0
A second scene load, different arc	total went 277 → 554 — exactly 277 per boot
Walking 600+ ticks into fresh streamed chunks	0 additional
Playing to tick ~1400, enemies 8 → 71	0 additional

That is a one-time init-time burst, deterministic per scene load, not a per-spawn or per-frame leak. It also disproved the leading theory outright — the mode check was never broken; it correctly refuses to instantiate a VFX portal on WebGL. It simply never got a say, because ten VisualEffect components had been baked into an authored prefab that the mode check cannot reach, and rode into the scene with every pooled gate. Three gate pools at prewarm three: thirty of them.

The nastiest detail is one only a live measurement finds. All thirty were already inactive, and Unity logged them anyway — so deactivating them was never going to help. They had to be deleted. The fix was one prefab, no C# at all, 850 lines of dead YAML removed.

None of that is reachable from a screenshot, and none of it is reachable from unit tests. Getting there took counting a console burst at the main menu, then again after a scene load, then again after a second scene load, then walking six hundred ticks to prove the count did not grow. That is a measurement session, and it is the thing the bridge is actually for.

Three rounds of being wrong in a browser

The honest part of this story is that the bridge did not work when it shipped.

The C# side is unit-tested — 178 EditMode test methods across ten test files, pinning dispatch, stepping, cheat sequencing, replay truncation, and the exact JSON of every acknowledgement byte-for-byte. But the .jslib layer, the actual JavaScript-to-C# boundary, only executes in a real WebGL development build. It is structurally out of reach of every automated test in the project.

So it got vetted by hand, in a browser, three times.

#403 filedBridge ships (#425) + round 1+2 vetsRound 3 vetRun telemetry + replay viewer (#467)Replay platform (#574)Typed DTOs (#608)

July 10 → August 1, 2026. Every mark after the first is a merged pull request against CandyRush dev.

Round one found two blockers. Multi-level level-ups soft-locked the bridge entirely: the second queued picker was visibly open on screen while the snapshot reported no choices at all, and only a human mouse click could clear it. The starting weapon-select gate was invisible and undriveable, meaning an agent could not even begin a run. It also found that the documented wait recipe deadlocked, for a reason worth repeating: Chrome fully suspends requestAnimationFrame on a hidden tab, and an automation agent almost always drives with the tab hidden. Unity's own loop limps along on a fallback timer so the sim keeps ticking, but any rAF-based wait in page JavaScript never runs at all.

Round two found the determinism story broken at its foundation: the pinned seed did not survive a page reload, because it lived in C# memory that the WASM reload destroys. Pin 424242, reload, and the run booted 2328243829. The round-1 report had flagged that as unverified. Round two verified it as broken.

Round three measured the fix for the thing that made all of this painful: step(600) went from 21,500 ms to 152 / 210 / 311 ms across consecutive calls. Ten sim-seconds in a fifth of a second, with the tab hidden. It also proved a fully autonomous run — weapon select, three level-ups, and death, with zero calls to resume and zero clicks past Play.

Then a second-half re-vet found that weapon-select reporting had regressed, and that a re-entrant rewind could strand the run at a starting gate that had been resolved three hundred ticks earlier.

Twenty-five numbered fixes came out of those rounds — F1 through F16, then R3F1 through R3F9. The pattern in them is consistent and slightly humbling: almost every one is a case where the C# was correct, the tests were green, and the thing still did not work, because the failure lived in ordering, in timing, or in a browser behaviour no test harness models.

Receipts

PR	Merged	Delivered	Diff	Files
#425	Jul 11	The bridge — telemetry, control, seed pinning, export	+7,845/−189	101
#429	Jul 11	Rounds 2+3: F1–F16, R3F1–R3F9, browser-verified	+5,414/−224	50
#467	Jul 13	Run/gate/boss telemetry, spatial layer, replay viewer	+8,422/−126	120
#574	Jul 21	Replay platform: production capture, boot-from-recording	+4,899/−617	78
#608	Aug 1	Typed DTOs, one serializer config, golden-pinned wire	+1,916/−541	30
#611 + #613	Aug 1	VFX audit command, then 277 errors deleted	+95/−851	5

What it still cannot do

Stating these plainly is cheaper than letting someone discover them:

There are no fire, dash, or ability inputs. CandyRush is an auto-attack survivor; per-tick input is movement only. The always-inert input fields were deleted rather than left as decoration.
spawnEnemies only spawns types with a live pool in the current segment. The acknowledgement says queued:true either way — check enemiesTotal.
setHp clamps to at least 1. Zeroing the healthbar would skip the death flow and strand the sim half-dead. To die, take real damage.
A rewind does not survive a page reload — the pending reconstruction is process state. The persisted pin plus a manual export covers that case.
The .jslib boundary is still unverified by tests, permanently and by construction. That is the whole reason the browser vets exist.
The agent-mode/ path stub currently 404s on the deployed dev channel. It exists in the WebGL template but is not present in the build that is live as of this writing. Use ?agent=1, which works.

And one practical warning that has nothing to do with the bridge: the dev channel is a development WebGL build, and it is heavy. Loading it cold pulled 113.2 MB of asset data and a 21.9 MB wasm module. Content-Type is served correctly as application/wasm, so streaming instantiation works as intended — it is simply a large payload. Give it minutes, not seconds, and expect the tab to sit on "still waiting on run dependencies" for a good while before the bridge appears. Once it does, __candyStartRun gets you into a live run in two seconds.

The reason to build this rather than wire up a vision model is not really cost, though it is roughly an order of magnitude cheaper per decision. It is that a screenshot is a lossy render of state the engine already knows exactly, and every bug in the list above — the queued picker, the reload-eaten seed, the stranded weapon gate, the 277 unattributable errors — is invisible in a screenshot and obvious in a snapshot.

The game was already deterministic. The bridge just stopped throwing that away.