Define UI tests with prompts or tasks

A UI test checks whether a person can complete a task in your app using its screens, such as signing in or checking out. In Ansight, you can describe the journey for an agent or run a saved task with fixed steps and assertions. Each run keeps a history you can inspect if it fails.

Tests are saved as YAML in your workspace. Supply prompt for an agent to navigate and verify a journey, or supply taskId without prompt to execute a saved TypeScript task directly. Both use the test runner’s target selection, app launch, session evidence, suites, and history.

Use a prompt test when completing the scenario requires observation or decision-making. Use a task-only test when the steps and assertions are already encoded in a task.

Create a test

Tests live beneath ansight/tests. A file’s path becomes its default test ID: ansight/tests/onboarding/complete.yaml becomes onboarding.complete.

Extract a test from a replay

In the local session player, select the relevant timeline range, choose Extract test or task, then select Ansight test. Enter the expected final-state assertions, generate a draft, review its YAML, validate it, and save it to ansight/tests.

The CLI provides the same extraction from a recorded interval:

ansight task extract <session-id> --start 12.5 --end 24.8 \
  --format test --workspace . --title "Complete checkout" \
  --assertion "The order confirmation screen is visible"

The extractor puts recorded actions and observable checks together in the test prompt. A separate validation block is optional. If no assertion is supplied, it suggests one from a newly visible final UI label when possible. Review the starting state, input values, and intended outcome before running the test; recorded input text is omitted from the YAML draft.

Create a test in source control, review it from the local player’s Tests area, or use the CLI. For a first test, name the journey and the final state you want to see:

ansight workspace add test . home.smoke \
  --app-id com.example.app \
  --prompt "Open the app, go to Home, and verify that the Home screen is visible."

With the app available on a connected device, run it with ansight test run . home.smoke. Replace com.example.app with your app’s bundle or package ID.

A more complete example

Add a name, specific assertions, and a secret alias when the journey needs them:

ansight workspace add test . onboarding.complete \
  --app-id com.example.app \
  --name "Complete onboarding" \
  --prompt "Launch the app and complete the onboarding flow." \
  --validation "Verify the signed-in home state through Ansight." \
  --assertion "The onboarding screen is no longer visible" \
  --assertion "The signed-in account name is visible" \
  --required-secret login.password

The generated definition is source-controlled YAML. Block text keeps longer journeys and validation instructions readable:

schemaVersion: 1
id: onboarding.complete
name: Complete onboarding
appId: com.example.app
hintTasks:
  - onboarding.open-sign-in
  - onboarding.submit-sign-in
prompt: |-
  Launch the app and complete the onboarding flow.
requiredSecrets:
  - login.password
validation:
  prompt: |-
    Verify the signed-in home state through Ansight.
  assertions:
    - The onboarding screen is no longer visible
    - The signed-in account name is visible

Workspace tests use schema version 1. Save each test as a .yaml file with a unique ID. Validation reports duplicate IDs instead of choosing one definition.

Write actions and observable checks together in prompt, in journey order. Use validation only when you prefer a separate checklist. YAML comments are supported. Use |- for multiline text and quote values that YAML might otherwise interpret as numbers or booleans. Test files must contain one YAML document.

Run a task without an agent

Omit prompt and set taskId to the exact path-derived ID of a saved workspace task. For example, ansight/tasks/checkout/complete.ts has ID checkout.complete:

schemaVersion: 1
id: checkout.smoke
name: Checkout smoke test
appId: com.example.app
taskId: checkout.complete

Run it with ansight test run . checkout.smoke, or include it in ansight test run-all .. The runner launches or attaches to the app and calls the task directly. It makes no model calls and needs no AI gateway preparation. The task’s timeout, action limits, capability requirements, and named assertions apply. Only a Passed task result passes the test; a task with no assertions is inconclusive and does not pass.

For a task that declares inputs, add an optional taskInput mapping using that task’s input schema. Values are checked before launch and schema defaults apply:

appId: com.example.app
taskId: search.verify-results
taskInput:
  query: Eagle Rock
  minimumResults: 1

Keep secret values out of inputs. Declare aliases under requiredSecrets and read them in the task with secrets.require(alias) or secrets.get(alias). Task-only tests reject validation and hintTasks; put checks in the task’s expect assertions. taskInput is only supported when prompt is omitted.

Existing definitions containing both taskId and prompt continue to run through the agent. Removing prompt opts into deterministic execution. A present but empty prompt is invalid. --execution-mode sdk|device selects the app connection independently of whether the test uses an agent.

Run without the Ansight SDK

Use --execution-mode device for UI prompt tests and portable tasks on Android devices and iOS simulators. A physical Android phone needs --platform android and its exact ADB serial in --device-id. The host captures screenshots and accessibility trees and drives device input without an Ansight SDK connection:

ansight test run . onboarding.complete --app /path/App.app \
  --device-id SIMULATOR-UUID --execution-mode device

Use a simulator .app for iOS or an .apk for Android. Portable tasks use task descriptor schema version 2 and declare their capability requirements; version 1 tasks retain their SDK expectation. External CPU, memory, native logs, and sandbox files are captured where available. Android supports renderer FPS through gfxinfo; iOS simulator FPS is currently unavailable. In-process remote tools and App Graphs require sdk mode.

An active app monitor on the selected device is reused when running an installed app without an application artifact. Tests can also attach directly with --session-id <live-session-id>; the capture determines the execution mode. The session is reserved for one execution at a time. Completing, failing, or cancelling an attached test releases the reservation and leaves the app and monitor running. Supplying an artifact requests installation and does not reuse the monitor’s device reservation.

Screenshots and UI evidence remain available in the recorded session. --headless controls device windows independently.

See External App Monitoring to register monitors, link a workspace, and inspect the evidence available on each platform. Use capability requirements in any tasks referenced by these tests.

Cloud runner

Paid feature: Running tests on a cloud runner requires a Startup or higher Ansight Cloud plan.

Register a machine as a cloud runner

Sign in to your Ansight account, then open Runner in the local player on the machine that will run jobs. Select an organisation where you are an admin or owner, enter a machine name, and register it. Choose Enable runner when the machine is ready to accept work. The panel shows whether the runner is enabled and lets you stop it. Runner controls are unavailable while signed out.

The machine needs the target simulator, emulator, or device and a compatible build for each job. Upload the build and workspace revision before queueing a test that needs them. The runner installs and launches the app; it does not compile the app from source.

Queue a remote runner job

In the portal runner form, choose Test or Custom prompt, select the app and build, and set App connection → SDK-less. The job uses executionMode: "device". Ansight SDK remains the default and requires SDK provisioning support in the build.

Use an iOS Simulator .app bundle or an Android .apk. Device mode still checks app identity and build compatibility, including simulator architecture or APK signature and emulator compatibility. A physical-device IPA cannot be used as a simulator build. For a test, also select an uploaded workspace revision and its test definitions. Build the app with your normal build tools before upload.

The runner’s Automated Task option runs saved tasks against already connected sessions. Open or externally monitor the app on the selected device first, and use a version 2 task when that session has no SDK provider.

Under Secrets, select only the aliases needed by the run. Save for team stores a value for that app and team; Use for this run supplies a one-run value. Required test aliases are included automatically, and missing values must be supplied before queueing. Keep aliases in test definitions. Task code reads granted values with secrets.get(alias) or secrets.require(alias); do not return or log them. These runner settings are separate from the local CLI’s app-scoped credential store.

Test fields

Checks in prompt are mandatory, just like checks in validation. The runner must verify each requested outcome with Ansight evidence at the point it is requested. Check transient confirmations before navigating away; a successful tap alone does not prove its expected result.

Use hintTasks when known repository tasks cover parts of the journey. The agent checks each hint’s starting state and scope before calling it, and uses its passed assertions as evidence instead of replaying the same steps. Hints do not replace the test’s requested checks. An unavailable hint does not stop the test; use --trace to see it in task-discovery diagnostics.

Platform, device-kind, and framework constraints belong to the referenced task, not the test and not its input arguments. Add optional platforms, deviceKinds, and frameworks arrays to the task descriptor, then set the test’s taskId. The CLI runner validates an explicit target before launch, uses single-value metadata to constrain automatic selection, and checks the connected session’s framework before starting execution. Incompatible runs are reported as skipped capability combinations.

Inline task and selector references

Use @task/ID to reference an exact workspace task and @selector/ID to reference an exact automation ID. The surrounding instruction says what to do and when. For example, assuming these task and automation IDs exist:

appId: com.example.app
prompt: |-
  Run @task/locations.open with location "Eagle Rock".
  Tap @selector/location-copy.
  Verify that the visible confirmation states:
  "Copied the location for 'Eagle Rock' to your clipboard."
  Tap @selector/profile-tab.
  Verify that the Profile screen opens.

In the test YAML editor, type @task/ for enabled tasks from the app’s workspace or @selector/ for automation IDs recorded in the selected replay period. Completion works inside the main prompt and optional validation text. Hover a task for its description and input schema, or a selector for its matching fields. Task references also support the existing task preview.

Task references are checked before execution and prioritized for task discovery. An unknown or disabled task is an error, and task references require workspace tools to be enabled. Supply inputs in the surrounding prose according to the task’s input schema.

@selector/profile-tab resolves to {"automationId":"profile-tab","matchMode":"exact"}. It does not refer to a saved alias or a screen label. An ID absent from recorded evidence is flagged for review; the live app must still contain the appropriate target when that step runs. Completion does not prove that the target is currently visible.

IDs containing spaces or special characters must be percent-encoded, for example @selector/Profile%20tab or @selector/com.example%3Aid%2Fcopy. Completion handles encoding automatically. A trailing sentence period is punctuation; encode a literal final period in an ID as %2E. Use \@task/abc in a YAML block scalar when the text is a literal example rather than a reference.

Validation forms

Use a string for a compact validation instruction:

validation: Confirm that the signed-in home screen is visible.

Use an object when the test has named observable outcomes:

validation:
  prompt: Inspect the final state through the live visual tree.
  assertions:
    - The home screen is visible
    - The account name is Matthew

When supplied, a validation object must contain a non-empty prompt, at least one non-empty assertion, or both. Write assertions about state the runner can actually observe through screenshots, the live visual tree, logs, databases, app tools, or other available Ansight evidence.

Use secrets without putting values in test files

requiredSecrets contains aliases such as login.email or login.password. The definition, agent prompt, progress stream, and run audit never contain the secret value.

At run time, Ansight resolves each alias from the app-scoped secret store first, then from an environment variable with the exact same name. A stored value takes precedence. Use portable environment-variable names such as TEST_USER_PASSWORD when you want this fallback; only aliases explicitly listed in requiredSecrets are read from the environment.

For the resident host, select the app in Tests and save an alias and value under Test secrets to keep it in the operating system credential vault. The agent can type an authorized value with ansight_type_secret, but it cannot read or reveal that value regardless of its source.

The CLI exposes the same app-scoped secret store:

ansight secret set com.example.app login.password
ansight secret list com.example.app
ansight secret remove com.example.app login.password

secret set uses a hidden prompt by default. For automation, pass --stdin or --from-env NAME; this copies the value into the secret store. You can instead set an environment variable whose name matches the required alias and run the test without calling secret set. CLI runs forward only values for aliases declared by the selected tests when a resident host handles the command. Local player runs can use variables inherited by the host process.

The run fails before model or device work begins when any declared alias is missing from both sources or has an empty environment value.

Run from the local player

1. Open Tests in the local player and select the active app workspace.
2. Choose the detected app and test definition.
3. Select a simulator or emulator, or reuse an already connected app session.
4. Optionally choose an .app, .apk, or simulator-compatible .ipa to install.
5. Review the scenario prompt and validation, or the selected task and input, then choose Run test.
6. Follow task assertions and tool calls in Execution. Prompt tests also show model passes.
7. Open Run history to inspect the completed audit.

The runner can start a selected stopped simulator or emulator, install an explicit artifact, launch the app, and wait for the exact App ID’s Ansight SDK session. It does not build the application. Without an artifact, the target must already contain an installable package for the test’s appId.

Run from the CLI

List and validate definitions before running them:

ansight test list .
ansight test validate .

Run one test using a matching connected session or automatic target selection:

ansight test run . onboarding.complete

Select, start, or install a target explicitly when needed:

ansight test run . onboarding.complete --device-id <device-id>
ansight test run . onboarding.complete --app /path/to/App.app
ansight test run . onboarding.complete --app /path/to/app.apk
ansight test run . onboarding.complete --ipa /path/to/App.ipa

Choose whether device windows open

The CLI shows simulator/emulator windows by default when launching a virtual target. For a windowless run, including in CI, pass --headless explicitly:

ansight test run . onboarding.complete --headless --json
ansight test run-all . --headless --json

The flag applies to every selected device in multi-device and --parallel runs, and to device starts requested by tools during the run. It is a runtime CLI option, not a field in the test definition. --json or --silent alone does not prevent windows from opening.

On iOS, the simulator boots without opening Simulator.app. Newly started Android emulators use -no-window. Existing windows are not closed, already running emulators are not restarted, and physical devices are unaffected. Reusing a connected session without relaunch leaves its window state unchanged. See headless device launches.

Select multiple devices and suite scheduling

Run the same test concurrently across an explicit device set by repeating --device-id (or its --device alias):

ansight test run . onboarding.complete \
  --device-id <ios-simulator-id> \
  --device-id <android-emulator-id>

Physical-target launches provision the SDK connection without a prior manual QR scan. The runner creates a transient, app-specific, one-use invite and injects its compact payload into the launched process. The app must opt in with WithUnattendedProvisioning() in .NET or withUnattendedProvisioning() in the native iOS and Android SDK builders. The payload is not included in command output or a public invite file; an unused invite is revoked after the session wait.

Run the suite, or a selected subset:

ansight test run-all .
ansight test run-all . \
  --test onboarding.complete \
  --test checkout.purchase \
  --device-id <ios-simulator-id> \
  --device-id <android-emulator-id> \
  --stop-on-failure

ansight test run-all . --parallel \
  --device-id <ios-simulator-id-1> \
  --device-id <ios-simulator-id-2>

For run-all, Ansight starts every selected device run for the current test as one concurrent cohort. It waits for the entire cohort to finish before starting the next test. --stop-on-failure still lets the current cohort finish, then skips later tests when any device run failed.

Pass --parallel when the selected devices should run different tests. Each selected test runs exactly once. Ansight initially assigns one test to each available device, then assigns the next pending test whenever a device finishes. If there are more devices than tests, the extra devices remain idle. With --stop-on-failure, active tests finish but no new pending tests are assigned after the failure is known.

run-all creates and maintains a managed batch run ID automatically. When tests must be launched independently—for example, by separate CI jobs—assign the same correlation ID to each test run invocation:

ansight test run . onboarding.complete --batch release_20260901
ansight test run . checkout.purchase --batch release_20260901
ansight test history --batch release_20260901
ansight test inspect release_20260901

Each execution keeps its own unique run ID. The supplied batch ID groups the retained audits without creating or overwriting a managed run-all audit. It may contain up to 128 letters, numbers, hyphens, or underscores.

Useful runner options include --session-id, --platform ios|android, --headless, --wait-seconds, --model, --max-turns, --max-round-trips, --max-tool-calls, --batch, --result-file, --audit, and --json.

The local player and CLI use the authenticated hosted test runner when the selected organisation permits workspace testing. Creating, editing, and validating test definitions locally does not require an account; running an agent-driven test does. Model credentials are issued through the hosted workspace execution flow.

Tasks available during a prompt test

By default, a test run connects compatible repository tasks from the same workspace and App ID. Before the first model turn, the host ranks tasks against the instruction and exposes the leading matches as task-specific tools with their declared input schemas. The agent can invoke one immediately or after an uncovered prefix reaches the task’s starting state. Focused task search remains available when the preloaded matches do not cover the goal.

Prompt tests do not need to list or duplicate those tasks in their YAML definitions. Use --no-workspace-tools when a CLI run must exclude repository tasks entirely. Repository triggers are also independent of test definitions; already-connected triggers continue reacting to matching events during a test.

Results and history

A complete run records the exact test, App ID, target and session, elapsed time, and final outcome. Task-only runs retain the task result and named assertions; --trace adds task source and tool-call evidence. Prompt tests also record the prompt, validation, model usage, and agent tool calls. All requested checks are mandatory, including those in the main prompt: the runner must verify them through Ansight evidence and fail when any assertion cannot be proven.

Use the local player Run history view for interactive review or the CLI for a machine-readable audit:

ansight test history --app-id com.example.app
ansight test history --batch release_20260901
ansight test run . onboarding.complete --audit --json

The local session explorer also links captures back to their run history. When the selected session has a matching recorded run, choose Trace to open that exact execution in Tests. The trace contains the task or agent execution, tool calls, arguments, results, evidence references, and final outcome retained for that run. Model passes and token usage apply only to prompt tests.

Local run history is free and remains on the machine. A paying or authorised organisation can track safe run summaries in the portal after registering its App ID:

ansight cloud app register com.example.app \
  --team-id <organisation-uuid> \
  --name "Example App"
ansight cloud test upload <run-id> \
  --team-id <organisation-uuid>
ansight cloud test sync com.example.app \
  --team-id <organisation-uuid> \
  --limit 100

Tracked summaries contain outcome, timing, counts, model, source type, and optional version/branch/commit context. Prompt bodies, validation bodies, secrets, logs, screenshots, and source files are not uploaded by these commands.

Replaying a saved run creates a new run. It preserves the recorded test instructions, platform, model limits, App ID, and secret aliases while resolving current secret values and a currently available compatible target.

Limits and troubleshooting

• A workspace can load up to 512 tests; each definition is limited to 1 MiB.
• requiredSecrets accepts up to 32 unique aliases.
• hintTasks accepts up to five unique repository task IDs.
• The runner refuses to target a connected app outside the active workspace.
• An IPA must contain a simulator-compatible Payload/*.app; a device-only IPA cannot run in Simulator.
• If more than one suitable target is available, specify one --device-id to disambiguate or repeat it to run against an explicit device set.
• If launch succeeds but the run times out waiting for a session, confirm the installed development build initializes the Ansight SDK and uses the expected App ID.
• Run ansight test validate . when a test is missing from the local player or CLI catalog.

Related docs

• Workspace Getting Started
• Tasks
• Trends
• Tooling and Validation
• Device Preview