Task API Reference

Reference for repository task descriptors, invocation types, host evidence APIs, app-tool suites, result envelopes, and named assertions.

Repository tasks use the generated ansight/tasks/ansight-task.d.ts contract. The copy generated by your installed CLI is authoritative for that version. The linked source and this guide provide an online reference.

Run ansight workspace init . to create or refresh the declaration in a new workspace. Existing customized support files are preserved unless --force is supplied.

Module shape

Every task exports a JSON-compatible descriptor and one default function:

import type {
  TaskDefinition,
  TaskInvocation
} from "./ansight-task.d.ts";

type Input = { automationId: string };
type Output = { matchCount: number };

export const task = {
  schemaVersion: 1,
  title: "Verify home",
  description: "Proves that the signed-in home screen is visible."
} satisfies TaskDefinition;

export default async function run(
  { run, input, ansight, app, expect }: TaskInvocation<Input>
): Promise<Output> {
  const result = await ansight.ui.assert({
    automationId: input.automationId,
    exists: true
  });

  expect(result.passed, { id: "home-visible" }).toBe(true);
  return { matchCount: result.matchCount };
}

TaskDefinition

FieldTypeRequiredContract
schemaVersion1 | 2NoDefaults to version 1, which requires an SDK app-tool provider. Use version 2 for capability-aware execution.
requiresExecutionRequirementsNoVersion 2 only: required capabilities and executable appTools IDs, checked before execution.
enabledbooleanNoDefaults to true. Disabled tasks remain discoverable and reject execution.
appIdstringNoExact optional app scope.
titlestringYesShort discovery label.
descriptionstringYesComplete workflow and authoritative result.
featurestringNoFocused discovery domain such as map.
keywordsstring[]NoAt most 32 discovery terms.
inputSchemaJsonSchemaNoValidates the object supplied as input and applies defaults.
outputSchemaJsonSchemaNoDescribes returned output; version 1 does not enforce it.
hostToolsstring[]NoExtra host tools callable through ansight.callTool; maximum 32. Standard typed APIs such as ansight.keyboard need no declaration.
timeoutSecondsnumberNoWhole run: 1–300 seconds; default 120.
maximumActionsnumberNoHost, app, discovery, and child-task calls: 1–100; default 64.

The descriptor must be a static object literal. Do not use functions, variables, spreads, or computed values inside it.

TaskInvocation<TInput>

PropertyTypeMeaning
runReadonly<TaskRun>Host-owned run identity and effective limits.
inputReadonly<TInput>Validated input after schema defaults are applied.
secretsget(alias), require(alias)Resolve only values granted to this run. get returns undefined for an ungranted alias; require throws.
ansightAnsightHostSession-pinned host evidence and control methods.
appTaskAppContextStandard app-tool suites and custom-tool bridge.
expectTaskExpectPlaywright-style named assertions that establish pass or failure.

TaskRun

PropertyTypeMeaning
runIdstringUnique run ID.
executionMode"sdk" | "device"The selected session’s capture and execution mode.
taskIdstringRepository-relative task ID.
appIdstringExact app scope enforced by the host.
sessionIdstringLive session enforced for every call.
repositoryRootPathstringAbsolute connected repository root.
timeoutSecondsnumberEffective whole-task timeout.
maximumActionsnumberEffective action limit.

ansight: host API

Host methods are bound to run.sessionId. Target fields supplied by task code are discarded and the selected session is injected.

Task descriptors can declare optional platforms, deviceKinds, and frameworks arrays. These are metadata—not task inputs. A workspace test can reference the task with taskId; the CLI runner then validates the metadata against its selected target and connected session before agent execution.

Platform-sensitive declarations carry custom JSDoc tags in ansight-task.d.ts: @supportedPlatforms, @unsupportedPlatforms, @supportedDeviceKinds, @unsupportedDeviceKinds, @supportedFrameworks, and @unsupportedFrameworks. TypeScript preserves these descriptive tags for editor hovers and declaration parsers. Framework values include dotnet-maui, dotnet-ios, dotnet-android, react-native, flutter, capacitor, ios-uikit, ios-swiftui, android-views, android-compose, and native-unknown.

NamespaceMethodsMain result contracts
ansight.capabilitiessnapshot(), require(requirements)ExecutionCapabilitySnapshot with mode, app-tool availability, providers, and reasons
ansight.fileslist(args), read(args), capture(args)External sandbox entries, base64 content, or retained snapshot metadata
ansight.spansmeasure(name, operation, options?)Return the operation’s result and emit host-observed span events
ansight.sessiongetAppState(), getProperties<T>(), getTimeline(args?)SessionAppStateResult, SessionPropertiesResult<T>, SessionTimelineResult
ansight.uifind(args?), waitFor(args?), assert(args?), tap(args?), typeText(args), swipe(args?), scroll(args?), pinch(args), back(args?), runSequence(args), getLiveVisualTree(args?), getLiveNavigationStructure(args?), getVisualTreeSnapshot(args?), searchVisualTree(args?)UiFindResult, UiWaitResult, UiAssertResult, UiActionResult, UiSequenceResult, visual-tree contracts
ansight.networkget(args?), getRequest(args), readBody(args)NetworkRequestsResult, NetworkRequestResult, NetworkBodyReadResult
ansight.logsget(args?), search(args), getContext(args?), getFacets(args?), getTimeline(args?), summarizeWindow(args?), extractExceptions(args?)Captured log, context, facet, timeline, summary, and exception contracts
ansight.artifactsgetNearest(args?), getSession(args?), listFiles(args?), readFile(args)Artifact descriptor and file contracts
ansight.screenshotstake(args?), getFrame(args?), assert(args?)Live capture envelope, persisted frame, and screenshot assertion
ansight.databaseassert(args)DatabaseAssertResult
ansight.telemetryget(args?), getTimeline(args?), summarizeWindow(args?)Telemetry samples, timeline, and summary
ansight.touchesgetTimeline(args?), getContext(args?), getGestureSegments(args?), findTapTargets(args?), getHeatmap(args?), findDead(args?), getArtifacts(args?), summarizeFlow(args?)Touch, gesture, heatmap, dead-touch, evidence, and flow contracts
ansight.annotationsget(args?), create(args), update(args), delete(args)AnnotationsResult, AnnotationMutationResult, AnnotationDeleteResult
ansight.appToolslist()AppToolListResult
ansight.taskslist(args?), run(args)TaskDiscoveryResult, TaskCallResult<TOutput>
ansight.lifecyclelaunch(), foreground(), background(), terminate()AppLifecycleResult
ansight.devicesetLocation(args), playLocation(args), clearLocation(), shake(args?), playAccelerometer(args), audioCapabilities(), injectMicrophoneAudio(args)Device location and motion, AudioCapabilitiesResult, AudioInjectionResult
ansight.permissionsnames, grant(args), revoke(args), query(args)PermissionResult with observed status and exact native mappings
ansight.permissions.iosnames, grant(args), revoke(args), query(args), reset(args)Native iOS service authorization on Simulator
ansight.permissions.androidnames, grant(args), revoke(args), query(args)Native Android manifest permission authorization
ansight.keyboardopen(args), isOpen(), dismiss(args?)KeyboardActionResult, KeyboardStateResult
ansightcallTool<TResult>(name, args?)Structured result of an extra declared host tool

For when to call each method, see Host API Methods. The generated declaration gives the exact argument and result shapes for your installed CLI version.

Native app permissions

Permission APIs operate on the task’s enforced application and native device in SDK or device mode. No hostTools declaration is needed. Await calls serially:

import { Permission, IosPermission, AndroidPermission } from "./ansight-task.js";

await ansight.permissions.grant({ permission: Permission.Microphone });
const access = await ansight.permissions.query({ permission: Permission.Microphone });
expect(access.status, { id: "microphone-granted" }).toBe("granted");
await ansight.permissions.revoke({ permission: Permission.Microphone });

// Use these within tasks for the corresponding platform.
await ansight.permissions.ios.reset({ permission: IosPermission.Photos });
await ansight.permissions.android.grant({ permission: AndroidPermission.Camera });

Named constants are exported by the generated ansight-task.js runtime module with readonly literal types in ansight-task.d.ts. Permission.Camera, IosPermission.LocationAlways, and AndroidPermission.RecordAudio provide completion and compile-time validation. Their matching type names can annotate variables. Strings and the lowercase .names catalogs remain supported, including app-defined Android identifiers.

The Permission API Reference documents every constant’s platform availability, exact native mapping, OS/tool restrictions, and links to the official Apple and Android documentation. Every constant also includes these details in its editor hover documentation.

PermissionResult reports supported, isSuccess, status, backend, nativePermissions, and the enforced session/app/device identifiers. Status is granted, denied, notDetermined, limited, unsupported, or unknown. Mixed native states produce limited; missing or unreadable state is unknown. Query returns unsupported results for unavailable resources. Unsupported mutations and native command failures throw. A mutation may change some native permissions before failing; its recorded result preserves individual states. Successful commands report the subsequently observed state, which may be unknown if the native store cannot be read.

Shared nameiOS serviceAndroid equivalent
cameraQuery via TCC; mutation only if listed by installed simctlCAMERA
microphonemicrophoneRECORD_AUDIO
contactscontactsREAD_CONTACTS
calendarcalendarDeclared READ_CALENDAR / WRITE_CALENDAR
photosphotosDeclared READ_MEDIA_IMAGES / READ_MEDIA_VIDEO on API 33+ with target SDK 33+; otherwise READ_EXTERNAL_STORAGE
locationlocation (when in use)Declared ACCESS_COARSE_LOCATION / ACCESS_FINE_LOCATION
locationAlwayslocation-alwaysACCESS_BACKGROUND_LOCATION on API 29+; foreground location on older versions
mediaLibrarymedia-libraryREAD_MEDIA_AUDIO on API 33+ with target SDK 33+; otherwise READ_EXTERNAL_STORAGE
motionmotionACTIVITY_RECOGNITION on API 29+
notificationsUnsupported by this simctl providerPOST_NOTIFICATIONS on API 33+

Android identifiers above have the android.permission. prefix. Mappings are resource approximations: Android contacts are read access, calendar and location use the permissions the app declares, photos cover declared visual media types, and media library covers shared audio. nativePermissions exposes the exact subset. On Android 14+, selected photo access is reported as limited and is also revoked by shared revoke({ permission: "photos" }). Background location grants first grant declared foreground dependencies; revocation removes background access only on API 29+. On iOS, revoking location-always denies location access entirely, following simctl behavior.

Android supports emulators and authorized physical devices through ADB, uses the active Android user, and requires declared permissions changeable through pm. Special access and app account roles are outside this API. iOS supports Simulator only. Native iOS service names also include photos-add, contacts-limited, reminders, and siri. Mutations use the installed simctl privacy service list; reset restores prompting on the next request. Queries read Simulator TCC and location authorization stores without changing them. Unrecognized store schemas return unknown. These describe app authorization, not global device switches or the ability of a particular hardware resource to function in Simulator.

Permission changes can terminate the running application. Subsequent calls still use the recorded session target; use ansight.lifecycle.launch() if the workflow needs to relaunch it. Native grants can bypass usage-description checks, so keep separate tests for the app’s actual permission request flow.

Execution capabilities

Use descriptor version 2 for tasks that support external sessions. Version 1 tasks retain their SDK requirement. Declare essential capabilities in requires so the host can refuse an incompatible session before running task code:

import type { TaskDefinition, TaskInvocation } from "./ansight-task.d.ts";

export const task = {
  schemaVersion: 2,
  title: "Capture notes database",
  description: "Save the simulator app's notes database to the session.",
  requires: { capabilities: ["files.capture"] }
} satisfies TaskDefinition;

export default async function run({ ansight }: TaskInvocation) {
  return await ansight.files.capture({
    root: "data",
    path: "Documents/notes.sqlite"
  });
}

That example uses an iOS data-container path. An Android app might instead store the file at files/notes.sqlite or databases/notes.sqlite.

await ansight.capabilities.snapshot() returns sessionId, executionMode, live, appAvailable, executable appTools, and a capabilities map. Each capability contains available, provider, and an unavailable reason. Use await ansight.capabilities.require({ capabilities: ["files.read"] }) for an optional branch that needs extra support. Available keys include ui.semantic, ui.screenshot, lifecycle, host.spans, evidence.read, evidence.logs, evidence.telemetry, evidence.artifacts, logs.native, files.read, files.capture, and app.tools; actual availability is session-specific.

app.available is false without a connected SDK provider. Calling any SDK app.* operation then rejects with code: "capability_unavailable", the missing capability, executionMode, and retryable: false. SDK-required operations include setup guidance. For a task that needs a particular app tool, declare requires: { appTools: ["files.list_directory"] } rather than assuming the suite exists. Missing declared requirements refuse the task; they do not skip the operation and report success.

External sandbox files

ansight.files uses the host’s external provider. SDK file tools remain under app.files. Require files.read for list or read, and files.capture for capture.

All three methods take a root-relative path and optional root (default "data"). An empty path lists the root. iOS also supports "app" and "group:<identifier>"; Android supports "data" only. Traversal and symlinks are refused. Android access needs run-as or an already provisioned root ADB daemon.

read returns base64 content and defaults to a maximum of 256 KiB; it cannot exceed that limit. capture defaults to 16 MiB and accepts maximumBytes up to 64 MiB, returning a retained snapshotId. Both return size, SHA-256, provider, timestamps, and consistency: "best-effort-live-copy". The automatic monitor’s exit-copy limit is separately fixed at 16 MiB per file.

Host-observed spans

ansight.spans.measure(name, asyncOperation, { group }?) brackets an operation and emits <name>.started, <name>.completed, or <name>.failed events. It uses two actions and requires host.spans. Use these events as Trends anchors for automation, or native log anchors for flows the app already logs.

Location route playback

ansight.device.playLocation(args) starts a GPX (or KML) route on the task’s live session device. Supply XML in routeContent, rather than a local path:

const playback = await ansight.device.playLocation({
  sourceFileName: "approach.gpx",
  routeContent: `<gpx version="1.1"><trk><trkseg>
    <trkpt lat="-35.094" lon="150.804" />
    <trkpt lat="-35.095" lon="150.805" />
  </trkseg></trk></gpx>`,
  mode: "fixed-speed",
  fixedSpeedKph: 20,
});
// Continue interacting with the app while the device moves.

The promise resolves after the first location is set and rejects if playback cannot start. The result includes runId, deviceId, platform, pointCount, and the effective timing settings. The device must be a booted simulator or emulator.

The default mode: "recorded" uses track timestamps. speed scales recorded timing from 0.1 to 100 (default 1). Without timestamps, playback falls back to fixedSpeedKph (0.5–500, default 30); speed does not scale fixed speed. loop: true repeats the route until stopped or replaced.

Playback shares the CLI’s single host-managed replay service. Starting a route replaces any active route, and playback continues after the task ends. Coordinate tasks that use location playback. Use ansight device location status to inspect progress or later failures, and ansight device location stop to stop it. clearLocation() clears the simulated coordinate but does not stop a running route.

Motion simulation

Use ansight.device.shake() to post a UIKit shake gesture to the task’s iOS Simulator, or send alternating accelerometer pulses to its Android Emulator. On Android, set intensity (5–50 m/s²), repetitions (1–8 pulse pairs), and intervalMs (40–500 ms per pulse) when the default shake is unsuitable:

await ansight.device.shake({ intensity: 24, repetitions: 3, intervalMs: 90 });
// Assert the app's shake response separately.

On iOS Simulator, call shake() without options. The gesture is fixed and received through UIKit’s responder chain. Ansight posts it to the selected Simulator through CoreSimulator; it does not inject Core Motion sensor samples.

For a controlled motion trace, use playAccelerometer. Each value is held for holdMs before the next sample. Values are in m/s², each axis is limited to −100–100, each hold is 10–1000 ms, and a sequence can have 1–100 samples and last at most 10 seconds:

await ansight.device.playAccelerometer({
  samples: [
    { x: 0, y: 9.81, z: 0, holdMs: 100 },
    { x: 18, y: 0, z: 0, holdMs: 80 },
    { x: -18, y: 0, z: 0, holdMs: 80 },
  ],
});

Android calls finish after all samples have been sent and the emulator’s previous acceleration has been restored. The result reports the sample or gesture count, duration, target device, and delivery backend; it does not confirm that the app handled the motion. Android Emulator supports sensor injection through its console. Physical-device motion is not supported by these methods.

Microphone audio input

Complete the microphone setup checklist before running an audio task. iOS audio tasks require one selected simulator and serial execution of the entire recording workflow; the injection lock alone does not protect concurrent recordings. See audio concurrency restrictions.

ansight.device.audioCapabilities() inspects the task’s existing live session without playing audio. Both audio APIs reject with audio-parallel-execution-unsupported in parallel test batches (including multi-device matrix batches); the error is logged in test progress and the task call audit. Run audio tests serially on one device without --parallel.

On an allowed serial run, audioCapabilities() returns available, a diagnostic code and message, provider details, the shared route scope, and accepted format/operation limits.

ansight.device.injectMicrophoneAudio({ file, timeoutMs?, waitForMicrophoneMs? }) delivers a speech fixture to the session’s virtual microphone. The app must already be recording through its normal flow and have microphone permission. This method does not launch an app, press Record, or supply a file directly to a recognizer.

// The app in this task's pinned session is already recording.
const injection = await ansight.device.injectMicrophoneAudio({
  file: "fixtures/audio/quote.wav",
  timeoutMs: 30_000,
  waitForMicrophoneMs: 0,
});
expect(injection.status, { id: "audio-delivery-completed" }).toBe("completed");
// Assert the app's transcript or saved recording separately.
  • file is relative to run.repositoryRootPath. Absolute paths, traversal outside that root, and symlinks resolving outside it are rejected. The host validates and snapshots the exact bytes before delivery. This API rule does not make trusted repository TypeScript a filesystem sandbox.
  • Input is RIFF/WAVE PCM16, mono, 16 kHz, at most 15 seconds and 1 MiB. Compressed audio is not converted automatically.
  • timeoutMs defaults to 30000, accepts 100–60000, and is capped by the task’s remaining deadline. waitForMicrophoneMs defaults to provider preflight without waiting (0) and accepts 0–10000. Android checks guest microphone readiness; iOS at zero wait verifies its route only, so the app must already be recording. Unsupported or unhealthy readiness is reported explicitly.
  • Calls are serial and awaited. Injection counts as one task action and blocks through delivery and cleanup. Task cancellation/deadline cancellation stops the provider; partial or uncertain delivery is never retried automatically.
  • Success returns AudioInjectionResult with fixture SHA-256 and format, submitted frames, delivery timing/provider diagnostics, and a retained evidenceId. captureVerified and transcriptionVerified are false: provider delivery does not establish what the app recorded or recognized.
  • iOS requires a separately installed loopback driver, exactly one booted simulator, and a verified input route in the configured Xcode’s running Simulator UI. The provider currently requires English audio-input menus and Accessibility access. It serializes injections across the shared host route. Readiness observes a host input client where supported, not the target app’s private recording stream. Android requires an authenticated, healthy emulator microphone provider. Emulator 37.1.11 has an observed native injection race that can crash the emulator; readiness guards do not guarantee stability. Ansight does not restart or retry automatically. Physical devices are unsupported.

The same operation is available through ansight audio inject. These standard methods require no hostTools declaration and cannot target a session, app, or device outside the task’s enforced session.

UI selectors

UiSelector is shared by find, wait, assert, and UI actions:

interface UiSelector {
  nodeId?: string;
  automationId?: string;
  text?: string;
  role?: string;
  type?: string;
  ancestorAutomationId?: string;
  action?: string;
  visible?: boolean;
  enabled?: boolean;
  exact?: boolean;
  caseSensitive?: boolean;
  index?: number;
}

Use ansight.ui.assert for evidence-producing expectations and waitFor for a bounded state transition. UiAssertArguments additionally supports expectedCount, expectedText, expectedValue, expectedVisible, expectedEnabled, and an optional actionId.

tap, typeText, swipe, scroll, pinch, and back persist before-and-after evidence and return UiActionResult. runSequence accepts from 1 through 32 already-known coordinate tap, swipe, pinch, or back actions and returns per-step outcomes plus sequenceEndpoints evidence. Use separate actions when the next step depends on an intermediate screen or when the evidence trail needs a checkpoint around every input.

The includeScreenshot action option controls whether the underlying host result contains final image content. Repository task methods receive structured action and evidence data. Set includeScreenshot: false in a task when inline image transport is unnecessary.

Software keyboard

ansight.keyboard.open(args) accepts the same stable selector fields as other targeted UI actions. It focuses the selected text input and verifies the opened keyboard through fresh device accessibility evidence. isOpen() performs a fresh read without input. dismiss(args?) is idempotent: it sends the platform back or Escape input only when device accessibility first confirms that the keyboard is open, then verifies that it closed. This guard prevents a dismiss call made against an already-closed keyboard from navigating away from the current screen.

Keyboard state is intentionally unavailable when the host cannot capture a device accessibility hierarchy that reports system keyboard presence. The methods fail explicitly in that case instead of inferring visibility from a focused text field or returning an unverified success.

Time windows

Network, log, telemetry, touch, annotation, artifact, and timeline methods accept their typed argument contracts from ansight-task.d.ts. Time-bound contracts use ISO-8601 UTC timestamps and bounded result limits. Prefer typed methods over ansight.callTool because they inject the task session and return stable named contracts.

Captured network requests

ansight.network inspects HTTP capture records already received by the task’s session. Its standard methods require no task.hostTools declaration. Calls must be awaited serially, like other host methods.

const failures = await ansight.network.get({
  methods: ["POST"],
  query: "/checkout",
  failedOnly: true,
  limit: 20
});

const first = failures.requests[0];
if (first) {
  const detail = await ansight.network.getRequest({ requestId: first.id });
  const response = await ansight.network.readBody({
    requestId: first.id,
    side: "response",
    maxBytes: 16 * 1024
  });

  // Header arrays preserve duplicate names; body content stays separately bounded.
  console.log(detail.request.responseHeaders, detail.headersTruncated);
  if (response.body !== null) {
    console.log(response.body.encoding, response.body.data);
  }
}

get accepts inclusive request-start startUtc/endUtc bounds, methods, statuses, host, query, failedOnly, limit, and cursor. Different filters combine with AND; values within methods or statuses combine with OR. Empty arrays impose no restriction. Methods, hostname substrings, and text queries are case-insensitive. Statuses accept integer codes from 100 through 599 or "1xx" through "5xx". query searches URL, method, and error message. failedOnly: true includes transport errors and HTTP statuses of at least 400; false imposes no restriction. Invalid dates, reversed time ranges, and invalid filters reject the call.

Pages contain complete summaries without header values or body content, ordered newest first with a request-ID tie-breaker. The default limit is 200 and the maximum is 1,000; the output budget can reduce the actual page size. Continue with get({ cursor: page.nextCursor }) after checking nextCursor is non-null. Cursors retain the initial matching snapshot, so live arrivals do not shift subsequent pages. The same filters may be repeated, but conflicting filters, a different session, or an expired cursor reject. Snapshots expire five minutes after creation and may be evicted earlier by the bounded host cache (32 snapshots, 16 Mi characters of summary data). After expiry or eviction, start a new query without a cursor; this also includes new arrivals. A matching snapshot that exceeds the cache budget rejects with a request to narrow the filters. matchedRequestCount describes the snapshot and isTruncated means matching records remain after the page.

getRequest rejects unknown request IDs and returns metadata plus bounded request/response header arrays. headersTruncated reports omitted complete entries. readBody rejects unknown request IDs, but returns body: null when the selected request has no retained body. A captured empty body instead has data: "" and returnedByteCount: 0. Missing capture data does not explain why content is unavailable.

maxBytes bounds decoded bytes, defaults to 16 KiB, and accepts 1 through 64 KiB. Reads return a prefix that preserves UTF-8 boundaries; binary content uses Base64. encoding identifies the representation, while capturedBytes, totalBytes, and returnedByteCount distinguish capture metadata from this read. truncated means capture omitted content; isTruncated means the read omitted retained content. Both must be false for a complete body. Capture sanitization is preserved, and body content is not automatically parsed as JSON. Metadata, headers, and body previews are subject to the host’s serialized output budget; an individual summary that cannot fit rejects explicitly.

Empty results do not prove capture is enabled or that an in-flight request has completed. These methods inspect retained traffic; they do not initiate, replay, intercept, or mock requests. Correlate network records with other evidence using ansight.session.getTimeline({ categories: ["networkRequest"] }).

Timeline and screenshot annotations

An annotation has a timeline point or range and may include screenshot shapes and a semantic visual-tree target. ansight.annotations.get(args?) keeps its existing time, label, frameId, and hasGeometry filters. create, update, and delete operate within the same enforced session and require no extra host tool declarations.

const created = await ansight.annotations.create({
  label: "Checkout failed",
  startUtc: failedRequest.startedAtUtc,
  endUtc: failedRequest.completedAtUtc,
  notes: `${failedRequest.method} ${failedRequest.url}`,
  geometries: [{
    kind: "rectangle",
    frameId: inspectedFrame.frameId,
    x: 0.1, y: 0.65, width: 0.8, height: 0.12
  }]
});

await ansight.annotations.update({
  annotationId: created.annotationId,
  expectedSource: created.annotation.source,
  notes: "Response body reviewed.",
  endUtc: null
});

// Remove the annotation if the task no longer needs it.
await ansight.annotations.delete({ annotationId: created.annotationId });

The example assumes the request and screenshot were inspected earlier in the task; its coordinates are illustrative. Derive real shapes from the selected frame. Annotation writes preserve evidence but do not establish task success; use a named expect assertion after writing failure evidence.

create requires a non-whitespace label; it accepts optional annotationId, notes, source, startUtc, endUtc, geometries, and target. It generates an ID when omitted and atomically rejects an existing explicit ID. The default source is task:<taskId>. An omitted start uses the session’s latest captured timestamp; supply an evidence timestamp for reproducible placement. A generated ID is not an idempotency key: reconcile an uncertain create before retrying.

update patches the latest saved record atomically. Omitted fields remain unchanged. null clears notes, endUtc, or target; geometries: [] clears shapes; a supplied geometry array replaces the entire array. Empty notes normalize to no notes. Source, capture group, custom data, attached evidence, and hook failures are preserved. The resulting end cannot precede its start, and equality normalizes to a point. Timeline edits preserve actual screenshot capture times. A no-op update succeeds and returns the saved record.

update and delete require an existing annotationId. Their optional expectedSource is an exact-match guard checked atomically with the mutation; a mismatch rejects without changing the record. This guard checks provenance, not a revision. Concurrent patches preserve omitted fields; competing edits to a supplied field use the last write. Mutations return the saved normalized annotation, or deletedAnnotation after deletion. If the full saved or deleted record exceeds 480,000 serialized characters, the call rejects before mutating state. Existing host and CLI annotation operations remain available for large legacy records.

Geometry inputs are typed point, rectangle, ellipse, or freeDraw records, anchored by frameId. Coordinates are finite and normalized from 0 to 1 over the full screenshot. Boxes need positive dimensions and must fit within the image; free-draw paths need at least two points. Invalid geometry and styles reject. The host resolves the frame’s capture timestamp and derives free-draw bounds. Optional geometry fields are geometryId, text, strokeColor, and strokeWidth. Stroke colors accept #RRGGBB or #AARRGGBB; stroke width must be finite and positive.

A semantic target is { visualTreeSnapshotId, nodeId } from an inspected, retained visual-tree snapshot. The host resolves its label, type, automation ID, and bounds, and returns the node identity as target.targetId. Both frames and trees must belong to the enforced session; references never silently capture new evidence. Read results provide named geometry, target, inferred-shape, and attached-evidence types. Output geometries also describe legacy line and arrow shapes created by other host clients.

Task composition

ansight.tasks.list(args?) searches the same repository and App ID as the calling task. It accepts query, feature, and maxResults; focused queries use the same metadata, input-schema, synonym, and typo matching as ansight task list.

Use the exact discovered ID to invoke another task against the same live session:

type ChildOutput = { selectedArea: string };

const catalog = await ansight.tasks.list({
  query: "select map area",
  feature: "map"
});
const selected = catalog.tasks.find(candidate => candidate.taskId === "map.select-area");
if (!selected) throw new Error("The map selection task is unavailable.");

const child = await ansight.tasks.run<ChildOutput>({
  taskId: selected.taskId,
  input: { area: "Secret Garden" }
});
expect(child.output?.selectedArea, { id: "area-selected" }).toBe("Secret Garden");

run resolves only when the child task is Passed; every other child status rejects the call and therefore fails the caller unless its code deliberately handles the error. The child keeps its own run ID, assertions, timeout, action budget, tool-call audit, and persisted result. The parent records ansight.tasks.run as one action, and its remaining deadline caps the complete child call. Parent and child assertions are not merged, so the parent must still record at least one named assertion for its own outcome.

The host reloads the caller’s exact workspace for discovery and execution, pins every child to the caller’s App ID and session, rejects self-calls and recursive cycles, and permits at most eight task execution levels in one call chain. A child cannot select another repository, app, session, or device.

app: live app API

Every standard app method returns Promise<AppToolCallResult<TResult>>. The host pins the request to the task session, while the connected app enforces tool availability, grants, argument schemas, and critical-operation policy.

NamespaceMethods
app.artifactsquery(args?), request(args)
app.uigetVisualTree(args?), getScreenshot(args?), inspectNode(args?), showOverlay(args?), getOverlay(args?), queryOverlays(args?), updateOverlay(args?), removeOverlay(args?), clearOverlays(args?)
app.fileslistDirectory(args?), readFile(args?), getChecksum(args?), download(args?), beginBinaryDownload(args?), push(args?), copy(args?), move(args?), delete(args?)
app.fileDescriptorslistOpen(args?), countOpen(args?), inspect(args?), getUsage(args?)
app.jniReferencescaptureGraph(args?)
app.preferenceslistKeys(args?), get(args?), set(args?), remove(args?)
app.clipboardgetText(), hasText(), setText(args), clear()
app.secureStorageget(args?), set(args?), remove(args?)
app.datalistDatabases(args?), describeSchema(args?), query(args?)
app.reflectionlistRoots(args?), inspectObject(args?), describeType(args?), setMemberValue(args?), invokeMethod(args?)
app.mauigetCurrentPage(args?), getVisualTree(args?), findElements(args?), getElement(args?), getBindableProperty(args?), setBindableProperty(args?), clearBindableProperty(args?), inflateXaml(args?), addElement(args?), removeElement(args?), setAppTheme(args?), getBindingContext(args?), getBindings(args?), getResourceState(args?), getNavigationState(args?), invokeElementAction(args?), waitForUi(args?), getLayoutDiagnostics(args?), getHandlerDiagnostics(args?), invokeBindingContextCommand(args?), setBindingContextProperty(args?)
app.reactgetComponentTree(args?), getShadowTree(args?), findComponents(args?), getComponent(args?), getNavigationState(args?), invokeComponentAction(args?)
app.fluttergetWidgetTree(args?), inspectWidget(args?), findWidgets(args?), getNavigationState(args?)
app.capacitorgetDocument(args?), inspectNode(args?), querySelector(args?), invokeAction(args?)
appcallTool<TResult>(toolId, args?) for an app-defined tool

app.clipboard uses the native Clipboard suite registered in the connected app. Reads require Read policy; set and clear require Write policy. Text is limited to 65,536 UTF-8 bytes, and iOS can show its normal paste-permission prompt. See Clipboard Tools for platform behavior and registration.

Standard suites require the corresponding SDK tool registration. A method can exist in the TypeScript contract while being unavailable in the selected app or denied by its runtime guard. The App Tool API explains when to use each method and how task calls differ from trigger action factories.

AppToolCallResult<TResult>

App results use a discriminated envelope:

type AppToolCallResult<TResult> =
  | {
      responseType: "tool.result";
      sessionId: string;
      appId: string;
      toolId: string;
      payload: {
        success: true;
        toolId: string;
        result: TResult;
        message?: string | null;
      };
    }
  | {
      responseType: "tool.error";
      sessionId: string;
      appId: string;
      toolId: string;
      payload: {
        success?: false;
        code: string;
        message: string;
        retryable: boolean;
        details?: JsonValue | null;
      };
    };

The full envelope can also contain host-applied defaults, associated artifacts, and persisted-visual-tree metadata. Narrow on responseType before reading the typed success result.

expect: named assertions

MatcherBehavior
expect(actual, { id, message? }).toBe(expected)Uses Object.is, matching Playwright’s strict identity matcher.
.toEqual(expected)Uses strict recursive deep equality.
.toBeTruthy() / .toBeFalsy()Checks JavaScript truthiness.
.toBeDefined() / .toBeUndefined() / .toBeNull()Checks nullish states explicitly.
.toContain(expected)Checks string inclusion or direct array membership.
.toContainEqual(expected)Checks deep-equal array membership.
.not.<matcher>Negates the following matcher.
expect.soft(actual, metadata).<matcher>Records a failure and continues the task function.

Assertion IDs must be stable and unique. At least one assertion must pass and none may fail for the task status to be Passed. Hard failures stop the task function; soft failures continue. A recorded failure remains authoritative even if task code catches the hard assertion error. Returning normally without an assertion produces Inconclusive.

Execution rules

  • Await host and app calls serially; overlapping calls are rejected.
  • Standard host APIs need no hostTools declaration.
  • ansight.tasks.list and ansight.tasks.run need no hostTools declaration.
  • ansight.callTool accepts only names declared in task.hostTools.
  • app.callTool targets app-defined tools and needs no descriptor declaration.
  • Every host, app, discovery, and child-task call counts toward the caller’s maximumActions.
  • Tasks can access only the enforced live session through Ansight APIs.
  • Repository TypeScript is trusted local code, not a filesystem or network sandbox.