How to debug an AI-generated Maestro test

Why these tests fail

An agent writes a plausible flow. It has read your source, so the steps are usually in the right order. What it cannot read is the rendered UI — and selectors are claims about exactly that. So the logic is fine and the selectors are guesses.

Two guesses go wrong in different ways, and they need different fixes:

• A wrong selector. The element is on screen, but the test names it wrong — a label that changed, a text match that hits three elements, an id that does not exist. Fixable by reading the hierarchy.
• A wrong premise. The screen the test is waiting for is not part of the app. No selector change fixes this, and this is the one that wastes afternoons.

Step 1 — Read the UI hierarchy

When a flow fails, Maestro writes debug artifacts and prints the directory at the end of the run. Inside is a view hierarchy of the screen where it stopped: every accessibility label and identifier that was actually present. This is the ground truth your selector was guessing at, and it is the first thing to read — before the YAML, before the logs.

Here is a real one. The test asserted a sign-in screen; this is what the app was actually showing:

accessibilityText: "TravelMind"
accessibilityText: "TripMind"
accessibilityText: "Your AI-powered travel companion"
accessibilityText: "Everything you need for the perfect trip"
accessibilityText: "AI Planning"          bounds: [33,674][119,693]
accessibilityText: "Explore"              bounds: [172,674][229,693]
accessibilityText: "Personal"             bounds: [292,674][359,693]
accessibilityText: "We'll need your permission for:"
accessibilityText: "Notifications"        id: "bell"
accessibilityText: "Location Services"    id: "location"
accessibilityText: "Start Your Journey"   bounds: [24,830][378,886]

No Sign In. No Email. No Password. The app opens onto an onboarding screen whose primary action is Start Your Journey.

Read the snapshot before you theorise. If it lists an id, use the id — never invent a coordinate tap for an element the hierarchy has already named.

Step 2 — Read the step timings

Durations are the second free signal, and the one most people skip. From the same run:

0. defineVariables        → COMPLETED (0.0s)
1. applyConfiguration     → COMPLETED (0.0s)
2. launchApp              → COMPLETED (2.0s)
3. assertCondition "Sign In" → FAILED  (17.2s)

Seventeen seconds on an assertion is not a slow screen. Maestro retries a selector until it times out, so that number is the retry window — the element never appeared at all. Asserts that pass typically land near 0.1s.

The rule worth internalising: a long duration on a failing step means wrong element, not slow app. Reaching for a longer wait there buys you the same failure, later.

Step 3 — Watch out for the plausible fix

Here is where this gets interesting, and where most write-ups stop short.

Given that evidence, the obvious fix is: the app opens onto onboarding, so tap through it first, then assert Sign In. That diagnosis is correct as far as it goes, and the resulting YAML looks well-grounded:

appId: com.example.travelmind
---
- launchApp:
    clearState: true
- assertVisible:
    text: "Start Your Journey"
- tapOn:
    text: "Start Your Journey"
    waitToSettleTimeoutMs: 500
- assertVisible:
    text: "Sign In"        # ← nothing has ever seen this screen
- tapOn:
    text: "Sign Up"

Run it and it gets three steps further, then dies on the same assertion. The app goes straight from onboarding to the home screen. It has no authentication at all — grep -r "Sign In" over the source returns nothing.

The fix corrected the step it had evidence for and preserved the assumption it did not. That is not a careless mistake; it is a structural limit. A hierarchy snapshot covers one screen — the one the flow stopped on. Everything a fix does after navigating away from that screen is unobserved.

Step 4 — Selector, or premise?

Before you accept any fix, ask which of the two you are looking at. The test is cheap:

• Does the element you are now targeting appear in the snapshot? Then it is a selector fix, and the evidence backs it.
• Does it live on a screen reached after the snapshot? Then it is a guess. Often a good one — but label it, and expect to be wrong sometimes.
• Does the snapshot contradict what the test set out to verify? Then stop editing YAML. Search the source for the screen. If it does not exist, the test is describing a flow the app does not have, and the honest fix is to change what the test is for.

That third case is common with generated tests, because the description they were generated from can be wrong about the app. The YAML is then a faithful rendering of a false premise — and it will absorb any number of selector fixes without ever passing.

Text selectors vs accessibility ids

When it is a selector problem, the durable fix is almost always the same: stop matching text, start matching ids.

A text selector matches the element's full accessibility label. That is why a tab labelled Scan and a heading that says Scan collide, and why a label carrying a counter — 5 scans left • Get Premium — breaks the moment the counter changes.

If the UI exposes no ids, add them

This is the honest prerequisite: you can only select by id if someone put one there. In SwiftUI that is one modifier.

Button("Get Premium") { showPaywall = true }
    .accessibilityIdentifier("premium_banner")

It costs a line and pays back on every future run — by the test, and by anyone reading the hierarchy afterwards.

Step 5 — Rerun and confirm

A fix is a hypothesis until the rerun passes. Watch the duration as well as the result: when a selector stops thrashing, the run gets shorter. One paywall flow we debugged this way went from failing at 49.2s to passing at 29.3s once the taps hit ids instead of text and an assertVisible gate replaced a blanket wait.

That second part matters as much as the ids. Gating a screen transition on assertVisible both synchronises and verifies, because it retries on its own. A blanket waitForAnimationToEnd after every tap does neither, and on a screen with an idle animation it burns its full timeout every time.

Where RunnerQA fits

Everything above works with Maestro alone. The tedious part is that the evidence is per-run, scattered across a temp directory, and gone by the time you need it — so most people debug from memory instead.

RunnerQA keeps that loop. Maestro still executes, on your own simulator or device through the CLI; RunnerQA holds the test catalog and run history, attaches the hierarchy and step timings each run produced, and hands that evidence to your agent so a proposed fix is grounded in what the run saw rather than what the YAML hoped.

Because of the limit in Step 3, an analysis also states how far the evidence carries it: grounded when every step is backed by the snapshot, assumed when a step targets a screen the evidence never showed, and premise unsupported when the evidence contradicts what the test set out to verify — in which case it says so instead of proposing YAML that cannot pass. Nothing is applied without your approval.

Honest limits

• The UI has to expose accessibility identifiers. If it does not, you add them first — no tool can select by an id that is not there.
• Runtime evidence tells you what the UI was, not what it was for. A human still decides whether the test or the app is wrong.
• Execution is local. RunnerQA is not a device farm and does not provide devices.
• Fixes are human-approved. This is not autonomous self-healing, and the Step 3 example is exactly why.
• Maestro is the executor. RunnerQA adds the catalog, the evidence, the history, and the approval — it does not replace it.

Questions