Livsy Code

Visual Verification: Giving AI Agents a Way to Check iOS Interfaces

Greetings, traveler!

A SwiftUI screen can compile, pass its unit tests, and still be difficult to use. A button may fall below the visible area on a small device. An error message may overlap the retry action at a larger text size. A loading indicator may remain visible after data arrives.

These defects appear when the application runs, often only under a particular combination of state and device settings. If an AI agent changes an interface, it needs a way to reach those conditions and observe the result. A build log cannot tell it whether the content is clipped or whether a navigation transition feels confusing.

With access to a running application, an agent can inspect its interface and produce material a developer can review. It works best alongside behavioral tests: screenshots show what was rendered at a particular moment, while interactions and assertions help establish how the application arrived there.

A screen with several states

Imagine an existing activity screen in a SwiftUI app. It displays a list of planned activities, an empty state when there are no items, and a retry action if loading fails. A change introduces a summary card above the list.

The task might sound like a simple layout update, but the card affects several situations:

  • With many activities, the list should remain scrollable and its final item reachable.
  • With no activities, the empty-state message and action should remain visible.
  • While data is loading, the screen should show its loading treatment without displaying stale content unintentionally.
  • If loading fails, the error and retry action should be readable and usable.
  • At accessibility text sizes, the summary should make room for larger labels.

Before asking an agent to verify the change, turn these expectations into concrete scenarios. Specify which state to open, which action to perform, and what would count as a failure. A screenshot of the populated screen is useful, but it cannot establish whether retry starts a new request or whether the empty state was checked at all.

The examples below describe a hypothetical application. The identifiers, fixture arguments, scheme, and paths are illustrative and need to be adapted to the project.

Make screen states reproducible

An agent cannot reliably inspect an error screen if it must wait for a real service to fail. The application needs a controlled route to each relevant state. Depending on the project, that route might use an existing mock service, a test-specific dependency container, or launch arguments that select local fixture data.

For example, a UI test could launch the app with these arguments:

let app = XCUIApplication()
app.launchArguments += [
    "-ui-testing",
    "-activities-scenario", "load-failure"
]
app.launch()

These arguments have no built-in meaning to iOS. The application must explicitly interpret them and configure its dependencies before the screen is created. A test configuration could inject a repository that returns a known failure for load-failure and a fixed collection for populated.

Keep this behavior confined to a test or debug configuration. Do not allow a production launch argument or an exposed deep link to bypass authentication or enable privileged test behavior. Use synthetic data and isolate any persistent state that tests change.

For the loading state, a test dependency can hold a request open until the test explicitly releases it. The loading indicator then remains visible long enough to inspect. The test can release the response and check the transition to the next state. If the chosen automation cannot coordinate that release, report a static loading capture separately from a tested transition.

A stable fixture should also control details that affect the image: dates, account names, image loading, locale, and the order of list items. When a screen depends on a clock, injecting a fixed time can be more reliable than trying to capture it at the same instant on every run. The goal is to make a failure reproducible without hiding behavior that the scenario is supposed to test.

Building and launching the app

An agent with access to Xcode’s command-line tools can build and launch an app on a specific simulator. These commands demonstrate setup rather than a complete visual test. They assume the simulator is already booted and require the project values to be replaced:

SIMULATOR_ID="YOUR-SIMULATOR-UDID"
APP_PATH="$PWD/.build/Build/Products/Debug-iphonesimulator/Activities.app"

xcodebuild \
  -workspace Activities.xcworkspace \
  -scheme Activities \
  -configuration Debug \
  -destination "platform=iOS Simulator,id=$SIMULATOR_ID" \
  -derivedDataPath "$PWD/.build" \
  build

xcrun simctl install "$SIMULATOR_ID" "$APP_PATH"
xcrun simctl launch "$SIMULATOR_ID" com.example.Activities \
  -ui-testing -activities-scenario populated

The build product path can differ with the project’s build settings. A script should confirm the app’s actual location before installing it. The example also assumes the app is not already running with a different scenario; terminate or reset it as appropriate before relaunching. A successful launch means the process was started, not that the desired screen has finished loading. A screenshot taken immediately after launch may show the launch screen or an intermediate state. The XCUITest example below waits for an expected element before capturing the app.

simctl is useful for simulator lifecycle operations, launching applications, and capturing screenshots or video. It is not a general-purpose UI testing framework. To tap controls, enter text, and wait for elements, use XCUITest or an automation tool that exposes those capabilities. A CLI or MCP server can give the agent a convenient interface to some of these operations, but the exact capabilities depend on the tool and its version.

An explicit simulator identifier matters when several devices are running. A command aimed at an unspecified booted device can capture a different simulator from the one used for the build or test.

Testing retry with XCUITest

A visual check becomes more useful when it is tied to an observable interaction. For the failure state, the test can confirm that the error is displayed and that retry leads to a populated list. The fixture must be designed to fail the first request and return the fixed list on retry.

The SwiftUI view can expose stable identifiers on the relevant elements:

Button("Try Again") {
    viewModel.retry()
}
.accessibilityIdentifier("activities.retry")

The corresponding test might look like this:

import XCTest

final class ActivitiesUITests: XCTestCase {
    func testRetryShowsActivities() {
        let app = XCUIApplication()
        app.launchArguments += [
            "-ui-testing",
            "-activities-scenario", "fail-then-succeed"
        ]
        app.launch()

        let retryButton = app.buttons["activities.retry"]
        guard retryButton.waitForExistence(timeout: 5) else {
            XCTFail("Retry button did not appear")
            return
        }

        let hittable = XCTNSPredicateExpectation(
            predicate: NSPredicate(format: "hittable == true"),
            object: retryButton
        )
        guard XCTWaiter.wait(for: [hittable], timeout: 5) == .completed else {
            XCTFail("Retry button was not hittable")
            return
        }
        retryButton.tap()

        let firstActivity = app.staticTexts["activities.item.morning-walk"]
        guard firstActivity.waitForExistence(timeout: 5) else {
            XCTFail("Activity did not appear after retry")
            return
        }

        let screenshot = XCTAttachment(screenshot: app.screenshot())
        screenshot.name = "Activities after retry"
        screenshot.lifetime = .keepAlways
        add(screenshot)
    }
}

Here, activities.item.morning-walk must be assigned to the text element in the fixture-backed screen. The test checks that a specific item appears after the tap. With the described fixture, that is evidence of the visible transition, but the assertion does not itself count requests or inspect their parameters. Check those through a controlled service or test dependency when the request is part of the acceptance criteria. The screenshot records how the app looked after the item appeared. Neither alone proves that the full layout is correct; a reviewer still needs to inspect the image or compare it against an appropriate reference.

An element can exist in the accessibility hierarchy without being currently hittable. The additional wait checks that XCUITest can find a point to tap; it does not establish that the control is easy for a person to use.

Prefer waiting for an expected element or state to sleeping for a fixed interval. A fixed delay can be unnecessarily long on a fast machine and too short on a busy CI runner. A timeout is still useful: it bounds the wait and makes a missing state visible as a failure. A test can pass even if the app briefly showed an incorrect intermediate state, so cover transitions separately when they matter.

Accessibility identifiers help automation locate elements, but they are not a substitute for accessibility labels. A control can be easy for a test to find and still be confusing to a VoiceOver user. Keep identifiers stable where practical and give accessible controls meaningful user-facing labels.

An inspection loop for the agent

Once the screen is reachable, the agent can follow a short sequence for each scenario:

  1. Launch the application with the selected fixture and device configuration.
  2. Navigate to the screen, using a supported test route or actual UI interactions.
  3. Wait for a condition that identifies the expected state.
  4. Perform the relevant action and check its observable outcome.
  5. Capture an image or video and record the configuration used.
  6. Inspect the result, investigate discrepancies, and repeat after a change.

The instructions should also state what to do when a step fails. If the agent cannot open the error state, it should report that scenario as blocked rather than using a screenshot from the populated state as evidence for both. If an image is ambiguous, it should describe what it could see and leave the uncertain point for review.

An MCP server can expose simulator interaction and screenshots as tools. XcodeBuildMCP and FlowDeck are examples of tools to investigate for this workflow; check the installed version for the operations it actually supports. Another agent may use a command-line wrapper or call XCUITest directly. The protocol or wrapper does not determine the quality of the check. What matters is whether the agent can select the right device, reach the intended state, act on the interface, and retrieve enough information to assess the outcome.

Some tools expose an accessibility hierarchy with element labels and frames. This can help an agent select a control more reliably than tapping a fixed coordinate. The hierarchy and the screenshot provide different information: an element may exist in the hierarchy while being obscured or awkwardly positioned on screen.

Comparing the result with a design

A design reference answers a different question from a regression baseline. A design describes an intended appearance; a baseline records a previously approved rendering. An implementation can match an old baseline and still differ from an updated design. It can also match a static design while a button does nothing.

When a design is available, compare equivalent conditions: the same screen, content, device dimensions, appearance, and text size. Ask the agent to inspect specific properties, such as whether the summary card fits above the first row, the empty-state action is visible, and the content respects the safe area. General instructions like “make it pixel perfect” are less useful when the design leaves dynamic content or accessibility sizes unspecified.

A side-by-side view is often enough for a human reviewer. An overlay or difference image can reveal shifts that are hard to spot between separate files. Take care when aligning the images: scaling, device frames, or cropping can introduce apparent differences or conceal a real one. Design tools and MCP integrations can provide access to reference frames and assets when configured, but the agent should still identify the exact reference and its state. A reference is not automatically a complete specification for every device or interaction.

Visual interpretation by a model is an aid to inspection, not a pixel-accurate measurement. Small typography changes, subtle contrast problems, and details outside a cropped image can be missed. Treat its observations as findings to check, especially when the interface carries financial, health, or other consequential information.

Visual regression and the cost of baselines

A visual regression test compares a new rendering with an approved reference. It is particularly useful for stable components and screens where an unintended layout shift would be expensive to discover later.

A typical workflow captures a reference image, reviews it, stores it with the test suite or in an artifact store, and compares subsequent runs with that reference. When a comparison fails, inspect both images and the difference before deciding whether to fix the code or approve a new baseline. Automatically replacing a baseline after every change turns the comparison into a record of the latest output rather than a meaningful regression check.

Choose the comparison method deliberately. Exact pixel matching can detect small changes but is sensitive to differences in rendering, fonts, operating system versions, and animation timing. Thresholds or perceptual comparisons may reduce noise, but they can also hide a meaningful defect. Define tolerances for the specific case and review what they exclude.

Pin the simulator model, runtime, appearance, locale, content size, and fixture data where possible. Separate baselines may be needed for materially different configurations. Dynamic areas can be stabilized or masked, but masking an area also removes it from the visual check. If the clock, avatar, or chart is relevant to the change, replace its input with deterministic data instead of masking it by default.

Snapshot tests of isolated SwiftUI views can run faster and cover more combinations. Full-screen captures exercise more of the running application, including navigation and container layout. They answer related but different questions, so choose based on the risk and the cost of maintaining each test.

Reviewing motion

A screenshot freezes one frame. It cannot show whether a sheet jumps during presentation, whether a loading indicator flickers, or whether content moves unexpectedly when the summary card appears.

For a focused animation review, start recording before the action and stop after the interface settles:

xcrun simctl io "$SIMULATOR_ID" recordVideo activity-transition.mp4

Perform the action while recording, then stop the command with Control-C. In an automated run, the process must be stopped cleanly after the scenario so the recording can be finalized. Recordings can be large, and simulator load or capture settings can affect what they show. They are useful evidence for reviewing a transition, not a reliable measurement of frame rate or responsiveness on a physical device. Use performance tools and representative hardware when performance is the question.

If the agent can inspect video, ask it to identify the beginning and end of the interaction and describe visible discontinuities. Otherwise, extract frames with timestamps for specific visual checks or ask a person to review the recording. Individual frames cannot establish whether motion was smooth. A person should review motion that affects comprehension or comfort, including the behavior with Reduce Motion enabled.

Accessibility and device variation

The default simulator configuration covers only one slice of the interface. A practical first pass for the activity screen might include a compact phone, a larger device, light and dark appearance, and at least one accessibility text size. Choose combinations based on the layout and the supported product, rather than generating every possible permutation.

At larger text sizes, inspect whether labels remain readable, actions stay reachable, and scrolling allows access to the whole screen. A screenshot can expose clipping, but it cannot tell whether VoiceOver announces a control correctly or whether its focus order makes sense.

XCUITest supports automated accessibility audits starting with Xcode 15 and iOS 17. Run the audit on a supported simulator or device; earlier iOS versions need other checks. For example, after navigating to the intended screen:

try app.performAccessibilityAudit()

Add this to a throwing UI test and run it on a supported simulator. Audit findings require investigation; a passing audit does not establish that every user can complete the flow. Check VoiceOver navigation and spoken labels manually, along with interaction patterns that need judgment. For changes involving haptics, performance, camera behavior, or hardware-specific interactions, include an appropriate physical-device check.

A report tied to the scenarios

A useful report lets a reviewer connect each claim to a scenario and an artifact. It does not need a complicated dashboard. A local HTML page or a pull request description can group screenshots by state and place the matching design or previous baseline beside each image.

For each result, record:

  • The scenario, expected behavior, and outcome: passed, failed, or not run.
  • The tested code revision, any uncommitted diff, build configuration, simulator model, and OS version.
  • Relevant settings such as locale, appearance, text size, and fixture.
  • The test result or image path, plus the exact point in the flow captured.
  • Any discrepancy, blocked step, or manual check still needed.

For example, an illustrative report could look like this:

Activities screen | revision: <commit> | working tree: <clean or diff reference>
Simulator: <model>, iOS <version> | English | Light | default text size

Populated: captured after first item appeared | artifacts/populated.png
Empty: captured after empty message appeared | artifacts/empty.png
Retry: UI test passed; item appeared after tap | artifacts/retry.png
Loading: not run; controlled pending response unavailable
VoiceOver: not run; manual review required

A visual report could display the images next to the corresponding design frames, labeled with their state and configuration. The example lists hypothetical outcomes, not results from a real test. A captured image is evidence for review; it should not be labeled a visual pass until the relevant comparison has been made. An automated audit on one state also does not establish that the whole screen has passed accessibility testing.

Generate the final report after the last code change and rerun checks affected by that change. An earlier screenshot can still be useful as a before image, but label it accordingly. Avoid placing customer data, authentication tokens, or confidential content in artifacts that will be shared or retained by CI.

Choosing the scope

Begin with the paths that are difficult to reproduce or easy to break. For the example screen, a fixture-backed retry test and captures of the empty and populated states may provide useful coverage without building a large visual testing system. Add a baseline when the screen has stable, reviewed output and the team is willing to maintain it.

Automation has a cost. Fixtures can drift away from real responses, screenshots require review, and operating system updates may invalidate baselines. Periodic integration checks against a representative backend help detect fixture drift. Real-device and manual reviews remain useful where simulator behavior or an automated assertion cannot answer the question.

The final report should identify the scenarios checked, the evidence available for each, and any work still needed before the change can be considered ready.

It might be interesting: