Get started

Bug-catch benchmark

How many seeded app bugs show up on Shipvo Phone, on Android and iOS: app logic, device hardware and Android and iOS only code, with the method and the command to run it.

Edit on GitHub

This page is for developers and teams who want to know how many app bugs show up on Shipvo Phone before a phone sees the build. You get the method, the measured result on Android and iOS, and the command to run it yourself.

What it measures

apps/bug-bench is an Expo app with 23 seeded bugs. Each bug has two versions of its screen, one with the bug and one with the fix, and one check. The check is a short script of taps, typing and screen reads, the same actions an agent uses.

A bug is caught when the check passes on the fixed screen and fails on the buggy screen, on Shipvo Phone. The benchmark opens every bug both ways on the Android phone and the iOS phone and counts.

ClassSeeded bugsExamples
App logic12The total ignores the quantity. A second tap places a second order. A slow older search replaces the newer result. Opening a screen throws while rendering.
Device hardware8A denied camera or location leaves a spinner. A failed fingerprint still pays. A declined or timed-out payment still places the order. A request made offline never finishes.
Android and iOS only code3The bug is in a .native.js file, in the native branch of Platform.select, or in a Platform.OS === "ios" branch.

The device hardware bugs are set up through the simulated device: the check sets the permission, the fingerprint result, the payment result or the network, then uses the app.

Result

Measured on 13 Nov 2026. A bug that exists on one platform only is counted on that platform only, so the 23 bugs give 45 checks.

The first row is from CI on a GitHub-hosted Linux runner: run 43383998094, job test, step pnpm -r test, which printed Total: 45 of 45 caught. The second row was measured in a development container (Linux, 4 CPUs, Node 22.23.4, Chrome Headless Shell) with BUG_BENCH_PHONE_PLATFORM=off node bench/run.mjs; the same container also measured 45 of 45 for the first row.

BuildAndroidiOSTotal
With phone platform mode22 of 2223 of 2345 of 45
Without phone platform mode20 of 2220 of 2340 of 45

Every app logic and device hardware bug was caught in both builds. The five checks that differ are the Android and iOS only code class: phone platform mode keeps that code in the simulator build and runs it as the phone the session simulates.

The benchmark is also the app's test. pnpm -r test builds the app, runs all 45 checks and fails if a seeded bug is missed or a check stops passing on the fixed screen.

Speed and size

apps/bug-bench/bench/speed.mjs times what an agent does all day on Shipvo Phone, and measures what the agent has to read. Measured on 13 Nov 2026 in a development container (Linux, 4 CPUs, Node 22.23.4, Chrome Headless Shell) with node bench/speed.mjs. Times are the median of 10 runs (5 for opening a phone). A second run gave the same numbers within 10 ms.

Pixel 8iPhone 16
Open a phone, to its first screen1,243 ms1,219 ms
Read the screen167 ms165 ms
Tap396 ms396 ms
Type388 ms380 ms
Screenshot250 ms250 ms
Screenshot of the phone in its frame370 ms369 ms
A screen as a tree (median of 5 screens)205 bytes205 bytes
The same screen as a PNG37,833 bytes39,830 bytes

A tap and a type include the wait for the screen to settle and the new screen in the reply. The tree is what an agent reads to act; on these screens the picture of the same screen is 185 times (Pixel 8) and 194 times (iPhone 16) its size.

Memory of the simulator and its browser, with phones held open: 589 MB with none, 936 MB with 1, 1,459 MB with 3, 2,016 MB with 5. Each extra phone adds 270 MB. A phone model adds nothing to that: the same run with no model, a Pixel 8 and an iPhone 16 gave 268.7, 268.3 and 268.9 MB per extra phone.

Two apps from GitHub

The same measurements on the first screen of two open-source Expo apps, built from their public source and opened with shipvo session. Same container, same day, median of 10 screen reads, 5 screenshots and 3 opens.

Obytes starter, Pixel 8Obytes starter, iPhone 16Bluesky, Pixel 8Bluesky, iPhone 16
Open a phone, to its first screen1,268 ms1,271 ms4,608 ms4,575 ms
Read the screen169 ms170 ms249 ms257 ms
Screenshot259 ms269 ms931 ms843 ms
Screenshot of the phone in its frame392 ms400 ms1,355 ms1,369 ms
The screen as a tree385 bytes385 bytes436 bytes436 bytes
The same screen as a PNG131,968 bytes151,204 bytes832,633 bytes827,438 bytes

Both apps were then used end to end: the Obytes starter (Expo 54, expo-router) through onboarding, form validation, login, its feed, a post screen, a phone restart on that screen, Settings, the theme sheet, dark theme and logout; Bluesky (Expo 58, React Navigation) through its welcome screen, the signed-out Discover feed, the drawer menu and the sign-in form.

Agent jobs

apps/bug-bench/bench/agent-jobs.mjs gives the work to a real coding agent. The agent gets one screen and one sentence on what the screen should do. It uses the screen through the Shipvo Phone MCP tools and ends its report with VERDICT: BUG or VERDICT: OK.

Each of the 8 jobs runs twice: on the screen with the seeded bug and on the fixed screen. The agent is not told which one is open. A run on the buggy screen passes when the verdict is BUG and the report names the bug. A run on the fixed screen passes when the verdict is OK, so an agent that always says BUG fails half its runs.

Measured on 14 Nov 2026 in a development container (Linux, 4 CPUs, Node 22.23.4) with Claude Code 2.1.325 and claude-sonnet-5-5, one run per job and variant, on a simulated Pixel 8. The agent had the phone tools only. Tokens are input, output, cache write and cache read together. Raw runs: bug-bench-agent-runs.json.

MCP server setupRuns passedBugs foundFalse alarmsTokensTool callsSeconds
As it ships16 of 168 of 80 of 83,157,77682191
Whole screens (SHIPVO_MCP_FULL_SCREEN=1)16 of 168 of 80 of 83,081,95981185.7
Core tools (SHIPVO_MCP_TOOLS=core)16 of 168 of 80 of 8993,65278187.9

What the numbers say:

  • The agent found all 8 seeded bugs and raised no false alarm, in every setup.
  • With SHIPVO_MCP_TOOLS=core the same jobs took 3.2 times fewer tokens: 993,652 against 3,157,776. The server then lists 14 tools, so the agent reads a much shorter tool list on every turn.
  • Changed lines and whole screens came out within 2.5% of each other. These screens have under ten lines, so a whole screen is already short.

The jobs are cart-total, stale-state, delete-wrong-row, double-submit, empty-form, pagination-repeat, nav-state-lost and button-stays-disabled. The heading of each screen shows its case id on both variants.

Run it yourself (needs the claude command signed in):

bash
pnpm --filter @shipvo/mcp... build
cd apps/bug-bench
BENCH_MODEL=claude-sonnet-5-5 node bench/agent-jobs.mjs --out runs.json

--jobs cart-total,empty-form picks jobs, --sides core-tools picks setups and --runs 3 repeats each run; tokens, tool calls and seconds are then the sum of each job's median.

Run another MCP server on the same jobs

--sides-file sides.json adds other MCP servers as setups, so two tools can be compared on the same jobs with the same agent and the same scoring. Each entry has a name, the serverName and MCP config to give the agent, and an openCmd that opens a case on that tool's device ({id} and {variant} are replaced):

json
[{ "name": "other-tool", "serverName": "phone",
   "config": { "mcpServers": { "phone": { "command": "npx", "args": ["-y", "other-tool", "mcp"] } } },
   "openCmd": "./open-case.sh {id} {variant}" }]

Run it

  1. Install and build the packages:

    bash
    corepack enable && pnpm install --frozen-lockfile
    pnpm --filter @shipvo/schema build && pnpm -r build
  2. Run the benchmark:

    bash
    cd apps/bug-bench
    node bench/run.mjs --out results.json
  3. Read the table it prints. results.json holds one row per check with fixedPass, buggyPass and result.

  4. For the speed and size numbers, run node bench/speed.mjs --out speed.json. --models "Pixel 8,iPhone 16 Pro" picks the phones and --skip-build reuses the build.

Options: --skip-build (use the existing sim-dist), --parallel <n> (checks at a time, default 4), --json (print the JSON instead of the table), --out <file>, --port <n>. Set BUG_BENCH_PHONE_PLATFORM=off to build without phone platform mode.

Each row's result is one of:

ResultMeaning
caughtThe check passes on the fixed screen and fails on the buggy one.
missedThe check passes on both. The bug did not show up.
invalidThe check fails on the fixed screen, or a run ended before its check answered. The command exits 1.

Add a bug

  1. Add a screen to apps/bug-bench/src/cases.js that takes buggy and register it under a new id.
  2. Add the same id to apps/bug-bench/bench/cases.mjs with its class, a title and a check.
  3. Run the benchmark. The new row must be caught or missed, never invalid.

Troubleshooting

You seeWhyFix
the web build failedThe Expo export stopped. The last lines of its output follow.Fix the named file, then run again.
the simulator did not startNo Chromium, or the port is taken.Run shipvo doctor, or pass --port <n>.
A row is invalidThe check fails on the fixed screen.Run with BUG_BENCH_DEBUG=1 to see why, and fix the check or the screen.

Next steps