Measured results

Lumi benchmark

Screens reached, flows finished and planted bugs found by Lumi on fixed apps, with the command to reproduce every number.

Edit on GitHub

This page is for developers who change Lumi or want to see what it finds. You will run one script and get a table of screens reached, flows finished and planted bugs found, the same every time for the same seed.

scripts/lumi/bench.mjs measures Lumi Autotest on a fixed set of apps with a fixed step budget. It reports three numbers per app and budget:

  • Screens reached: distinct screens the walker saw.
  • Flows finished: how many of the five flows (sign up, log in, search, add to cart, checkout) ended with a pass on valid input.
  • Planted bugs found: how many of the bugs switched on in the app showed up as a finding or a failed invalid-input flow.

Run it

  1. Build the CLI:

    bash
    pnpm --filter @shipvo/schema build && pnpm --filter @shipvo/cli build
  2. Run the bench:

    bash
    node scripts/lumi/bench.mjs                      # spec apps, budgets 50 and 200, seed 1
    node scripts/lumi/bench.mjs --budgets 100 --seed 3 --out bench.json
    node scripts/lumi/bench.mjs --start-sim --platform android   # also run Tide and Tackle on the Android simulator
    node scripts/lumi/bench.mjs --sim http://localhost:4546 --platform ios
  3. Read the output. The script prints a table and then the same data as JSON (--json prints only JSON). The same seed gives the same output. With the defaults it looks like this:

    text
    | app     | set      | budget | steps | screens | flows | bugs |
    | ------- | -------- | ------ | ----- | ------- | ----- | ---- |
    | shop    | tuning   | 50     | 43    | 5       | 2/5   | 3/3  |
    | shop    | tuning   | 200    | 44    | 5       | 2/5   | 3/3  |
    | account | held out | 50     | 50    | 0       | 2/5   | 1/3  |
    | account | held out | 200    | 71    | 8       | 2/5   | 3/3  |

Options:

OptionValuesWhat it does
--budgetswhole numbers, comma separatedStep budgets to run. Default 50,200.
--seedwhole numberSeed for the walker. Default 1.
--bugson, offTurn the planted bugs on or off. Default on.
--rankeron, offUse the trained ranker (see Lumi ranker). Default off.
--sim, --start-simURL, or no valueAlso run tide-and-tackle on a simulator.
--platformandroid, iosWhich simulator platform to use.
--json, --outno value, filePrint only JSON, or write JSON to a file.

Apps

AppSetRuns onPlanted bugs
shoptuningin-process spec appdead button, failing network call, validation that accepts bad input
accountheld outin-process spec appbroken route, blank screen, validation that accepts bad input
tide-and-tackletuningAndroid and iOS simulator (--sim or --start-sim)none

Held-out apps are for checking a change that was tuned on the others. Do not tune the walker on them.

How the budget works

A step is one tap, type, open-URL or back action. Opening a fresh phone is free. Flows run first, each with valid and then invalid input, and stop when the budget is used. The walker gets what is left. Walker replays do not count, the same as in shipvo explore --max-steps.

Planted bugs

Each bug has a flag in scripts/lumi/spec-app.mjs and a check that says which finding shows it. packages/cli/test/lumi-bench.test.ts turns on one bug at a time and checks that the engine finds that bug, and only that bug, with a budget of 1000. It also checks that nothing is found when all flags are off.

Results

Measured on 2026-11-07 with node scripts/lumi/bench.mjs --budgets 50,200 --seed 1 (bugs on, ranker off).

AppBudgetSteps usedScreensFlows finishedPlanted bugs found
shop504352 of 53 of 3
shop2004452 of 53 of 3
account505002 of 51 of 3
account2007182 of 53 of 3

Troubleshooting

You seeWhyFix
tide-and-tackle skipped: pass --sim <url> or --start-sim to run it on a simulator.The sample app needs a simulator, and you did not pass one.Add --start-sim or --sim <url>. The spec apps still run.
--ranker must be on or off, --bugs must be on or off or --platform must be android or iosThe value is not one of the listed ones. The script exits with code 2.Use one of the values in the options table.
--budgets must be whole numbers of 1 or more and --seed a whole number of 0 or moreA budget or the seed is not a whole number in range.Pass numbers such as --budgets 50,200 --seed 1.
--ranker on: no trained model in this buildNo model is built in.Run node scripts/lumi/train.mjs, then build again.
Different numbers from this pageA different seed, budget or bug setting.Use --seed 1 --budgets 50,200 --bugs on --ranker off.

Next steps