Workspace Tests

A workspace test describes an end-to-end user journey in JSON. Ansight starts or selects the target app, gives the scenario to a bounded agent runner, and requires the declared final state to be verified through Ansight tools.

Use a workspace test when completing the scenario requires observation or decision-making. For a fixed operation whose steps and tool calls are already known, use a task instead.

Create a test

Tests live beneath ansight/tests. A file’s path becomes its default test ID: ansight/tests/onboarding/complete.json becomes onboarding.complete.

Create a test from Studio’s Test workspace, write the JSON directly, or use the CLI:

ansight workspace add test . onboarding.complete \
  --app-id com.example.app \
  --name "Complete onboarding" \
  --prompt "Launch the app and complete the onboarding flow." \
  --validation "Verify the signed-in home state through Ansight." \
  --assertion "The onboarding screen is no longer visible" \
  --assertion "The signed-in account name is visible" \
  --required-secret login.password

The generated definition is ordinary source-controlled JSON:

{
  "schemaVersion": 1,
  "id": "onboarding.complete",
  "name": "Complete onboarding",
  "appId": "com.example.app",
  "prompt": "Launch the app and complete the onboarding flow.",
  "requiredSecrets": ["login.password"],
  "qualityChecks": ["onboarding-responsiveness"],
  "validation": {
    "prompt": "Verify the final signed-in home state through Ansight.",
    "assertions": [
      "The onboarding screen is no longer visible",
      "The signed-in account name is visible"
    ]
  }
}

Workspace tests use schema version 1. There are no legacy workspace-test formats.

Studio accepts comments and trailing commas when reading test definitions, but strict JSON is the most portable format for source control and other tooling.

Test fields

Keep action instructions in prompt and success criteria in validation. This separation makes the expected outcome easy to review and prevents a procedural step from being mistaken for proof.

Validation forms

Use a string for a compact validation instruction:

{
  "validation": "Confirm that the signed-in home screen is visible."
}

Use an object when the test has named observable outcomes:

{
  "validation": {
    "prompt": "Inspect the final state through the live visual tree.",
    "assertions": [
      "The home screen is visible",
      "The account name is Matthew"
    ]
  }
}

A validation object must contain a non-empty prompt, at least one non-empty assertion, or both. Write assertions about state the runner can actually observe through screenshots, the live visual tree, logs, databases, app tools, or other available Ansight evidence.

Use secrets without putting values in JSON

requiredSecrets contains aliases such as login.email or login.password. The definition, agent prompt, progress stream, and run audit never contain the secret value.

At run time, Ansight resolves each alias from the app-scoped secret store first, then from an environment variable with the exact same name. A stored value takes precedence. Use portable environment-variable names such as TEST_USER_PASSWORD when you want this fallback; only aliases explicitly listed in requiredSecrets are read from the environment.

In Studio, select the app in the Test workspace and save an alias and value under Test secrets to keep it in the operating system credential vault. The agent can type an authorized value with ansight_type_secret, but it cannot read or reveal that value regardless of its source.

The CLI exposes the same app-scoped secret store:

ansight secret set com.example.app login.password
ansight secret list com.example.app
ansight secret remove com.example.app login.password

secret set uses a hidden prompt by default. For automation, pass --stdin or --from-env NAME; this copies the value into the secret store. You can instead set an environment variable whose name matches the required alias and run the test without calling secret set. CLI runs forward only values for aliases declared by the selected tests when a resident host handles the command. Studio runs can use variables inherited by the Studio process.

The run fails before model or device work begins when any declared alias is missing from both sources or has an empty environment value.

Attach deterministic quality checks

qualityChecks contains IDs from ansight/quality. These numeric telemetry checks run after the functional scenario and before an app launched by the runner is stopped. Keep functional assertions in validation; use Quality for FPS, memory, duration, and other deterministic performance expectations.

Run from Studio

1. Open the Test workspace and select the active Agent workspace.
2. Choose the detected app and test definition.
3. Select a simulator or emulator, or reuse an already connected app session.
4. Optionally choose an .app, .apk, or simulator-compatible .ipa to install.
5. Review the scenario prompt and validation, then choose Run test.
6. Follow model passes and Ansight tool calls in Execution.
7. Open Run history to inspect the completed audit.

The runner can start a selected stopped simulator or emulator, install an explicit artifact, launch the app, and wait for the exact App ID’s Ansight SDK session. It does not build the application. Without an artifact, the target must already contain an installable package for the test’s appId.

Run from the CLI

List and validate definitions before running them:

ansight test list .
ansight test validate .

Run one test against the only matching booted target:

ansight test run . onboarding.complete

Select, start, or install a target explicitly when needed:

ansight test run . onboarding.complete --device-id <device-id>
ansight test run . onboarding.complete --app /path/to/App.app
ansight test run . onboarding.complete --app /path/to/app.apk
ansight test run . onboarding.complete --ipa /path/to/App.ipa

Physical-target launches provision the SDK connection without a prior manual QR scan. The runner creates a transient, app-specific, one-use invite and injects its compact payload into the launched process. The app must opt in with WithUnattendedProvisioning() in .NET or withUnattendedProvisioning() in the native iOS and Android SDK builders. The payload is not included in command output or a public invite file; an unused invite is revoked after the session wait.

Run the suite, or a selected subset:

ansight test run-all .
ansight test run-all . \
  --test onboarding.complete \
  --test checkout.purchase \
  --stop-on-failure

Useful runner options include --session-id, --platform ios|android, --wait-seconds, --model, --max-turns, --max-round-trips, --max-tool-calls, --result-file, --audit, and --json.

Studio and a signed-in CLI can use the authenticated hosted test runner when the selected organisation permits workspace testing. A CLI without an account can use a locally stored runner key configured with ansight secret openai set.

Tasks available during a test

By default, a test run connects compatible repository tasks from the same workspace and App ID. The agent can search for a focused task and invoke it when that is more reliable than reproducing the operation one UI action at a time.

Tests do not list or duplicate those tasks in their JSON. Use --no-workspace-tools when a CLI run must exclude repository tasks entirely. Repository triggers are also independent of test definitions; already-connected triggers continue reacting to matching events during a test.

Results and history

A complete run records the exact test, prompt, validation, App ID, target and session, elapsed time, model usage, tool calls, arguments, results, and final outcome. Required assertions are mandatory: the runner must verify them through Ansight evidence and fail when any assertion cannot be proven.

Use the Studio Run history view for interactive review or the CLI for a machine-readable audit:

ansight test history --app-id com.example.app
ansight test run . onboarding.complete --audit --json

Local run history is free and remains on the machine. A paying or authorised organisation can track safe run summaries in the portal after registering its App ID:

ansight cloud app register com.example.app \
  --team-id <organisation-uuid> \
  --name "Example App"
ansight cloud test upload <run-id> \
  --team-id <organisation-uuid>
ansight cloud test sync com.example.app \
  --team-id <organisation-uuid> \
  --limit 100

Tracked summaries contain outcome, timing, counts, model, source type, and optional version/branch/commit context. Prompt bodies, validation bodies, secrets, logs, screenshots, and source files are not uploaded by these commands.

Replaying a saved run creates a new run. It preserves the recorded test instructions, platform, model limits, App ID, and secret aliases while resolving current secret values and a currently available compatible target.

Limits and troubleshooting

• A workspace can load up to 512 tests; each definition is limited to 1 MiB.
• requiredSecrets accepts up to 32 unique aliases.
• The runner refuses to target a connected app outside the active workspace.
• An IPA must contain a simulator-compatible Payload/*.app; a device-only IPA cannot run in Simulator.
• If more than one suitable target is available, specify --device-id rather than relying on an arbitrary choice.
• If launch succeeds but the run times out waiting for a session, confirm the installed development build initializes the Ansight SDK and uses the expected App ID.
• Run ansight test validate . when a test is missing from Studio or the CLI catalog.

Related docs

• Workspace 101
• Tasks
• Quality
• Authoring and Validation
• Build Device Preview