Task API Reference
Reference for repository task descriptors, invocation types, host evidence APIs, app-tool suites, result envelopes, and named assertions.
Repository tasks use the generated ansight/tasks/ansight-task.d.ts contract.
The copy generated by your installed CLI is authoritative for that version.
The linked source and this guide provide an online reference.
Run ansight workspace init . to create or refresh the declaration in a new
workspace. Existing customized support files are preserved unless --force is
supplied.
Module shape
Every task exports a JSON-compatible descriptor and one default function:
import type {
TaskDefinition,
TaskInvocation
} from "./ansight-task.d.ts";
type Input = { automationId: string };
type Output = { matchCount: number };
export const task = {
schemaVersion: 1,
title: "Verify home",
description: "Proves that the signed-in home screen is visible."
} satisfies TaskDefinition;
export default async function run(
{ run, input, ansight, app, expect }: TaskInvocation<Input>
): Promise<Output> {
const result = await ansight.ui.assert({
automationId: input.automationId,
exists: true
});
expect(result.passed, { id: "home-visible" }).toBe(true);
return { matchCount: result.matchCount };
}
TaskDefinition
| Field | Type | Required | Contract |
|---|---|---|---|
schemaVersion | 1 | 2 | No | Defaults to version 1, which requires an SDK app-tool provider. Use version 2 for capability-aware execution. |
requires | ExecutionRequirements | No | Version 2 only: required capabilities and executable appTools IDs, checked before execution. |
enabled | boolean | No | Defaults to true. Disabled tasks remain discoverable and reject execution. |
appId | string | No | Exact optional app scope. |
title | string | Yes | Short discovery label. |
description | string | Yes | Complete workflow and authoritative result. |
feature | string | No | Focused discovery domain such as map. |
keywords | string[] | No | At most 32 discovery terms. |
inputSchema | JsonSchema | No | Validates the object supplied as input and applies defaults. |
outputSchema | JsonSchema | No | Describes returned output; version 1 does not enforce it. |
hostTools | string[] | No | Extra host tools callable through ansight.callTool; maximum 32. Standard typed APIs such as ansight.keyboard need no declaration. |
timeoutSeconds | number | No | Whole run: 1–300 seconds; default 120. |
maximumActions | number | No | Host, app, discovery, and child-task calls: 1–100; default 64. |
The descriptor must be a static object literal. Do not use functions, variables, spreads, or computed values inside it.
TaskInvocation<TInput>
| Property | Type | Meaning |
|---|---|---|
run | Readonly<TaskRun> | Host-owned run identity and effective limits. |
input | Readonly<TInput> | Validated input after schema defaults are applied. |
secrets | get(alias), require(alias) | Resolve only values granted to this run. get returns undefined for an ungranted alias; require throws. |
ansight | AnsightHost | Session-pinned host evidence and control methods. |
app | TaskAppContext | Standard app-tool suites and custom-tool bridge. |
expect | TaskExpect | Playwright-style named assertions that establish pass or failure. |
TaskRun
| Property | Type | Meaning |
|---|---|---|
runId | string | Unique run ID. |
executionMode | "sdk" | "device" | The selected session’s capture and execution mode. |
taskId | string | Repository-relative task ID. |
appId | string | Exact app scope enforced by the host. |
sessionId | string | Live session enforced for every call. |
repositoryRootPath | string | Absolute connected repository root. |
timeoutSeconds | number | Effective whole-task timeout. |
maximumActions | number | Effective action limit. |
ansight: host API
Host methods are bound to run.sessionId. Target fields supplied by task code
are discarded and the selected session is injected.
Task descriptors can declare optional platforms, deviceKinds, and
frameworks arrays. These are metadata—not task inputs. A workspace test can
reference the task with taskId; the CLI runner then validates the metadata
against its selected target and connected session before agent execution.
Platform-sensitive declarations carry custom JSDoc tags in ansight-task.d.ts:
@supportedPlatforms, @unsupportedPlatforms, @supportedDeviceKinds,
@unsupportedDeviceKinds, @supportedFrameworks, and
@unsupportedFrameworks. TypeScript preserves these descriptive tags for
editor hovers and declaration parsers. Framework values include dotnet-maui,
dotnet-ios, dotnet-android, react-native, flutter, capacitor,
ios-uikit, ios-swiftui, android-views, android-compose, and
native-unknown.
| Namespace | Methods | Main result contracts |
|---|---|---|
ansight.capabilities | snapshot(), require(requirements) | ExecutionCapabilitySnapshot with mode, app-tool availability, providers, and reasons |
ansight.files | list(args), read(args), capture(args) | External sandbox entries, base64 content, or retained snapshot metadata |
ansight.spans | measure(name, operation, options?) | Return the operation’s result and emit host-observed span events |
ansight.session | getAppState(), getProperties<T>(), getTimeline(args?) | SessionAppStateResult, SessionPropertiesResult<T>, SessionTimelineResult |
ansight.ui | find(args?), waitFor(args?), assert(args?), tap(args?), typeText(args), swipe(args?), scroll(args?), pinch(args), back(args?), runSequence(args), getLiveVisualTree(args?), getLiveNavigationStructure(args?), getVisualTreeSnapshot(args?), searchVisualTree(args?) | UiFindResult, UiWaitResult, UiAssertResult, UiActionResult, UiSequenceResult, visual-tree contracts |
ansight.network | get(args?), getRequest(args), readBody(args) | NetworkRequestsResult, NetworkRequestResult, NetworkBodyReadResult |
ansight.logs | get(args?), search(args), getContext(args?), getFacets(args?), getTimeline(args?), summarizeWindow(args?), extractExceptions(args?) | Captured log, context, facet, timeline, summary, and exception contracts |
ansight.artifacts | getNearest(args?), getSession(args?), listFiles(args?), readFile(args) | Artifact descriptor and file contracts |
ansight.screenshots | take(args?), getFrame(args?), assert(args?) | Live capture envelope, persisted frame, and screenshot assertion |
ansight.database | assert(args) | DatabaseAssertResult |
ansight.telemetry | get(args?), getTimeline(args?), summarizeWindow(args?) | Telemetry samples, timeline, and summary |
ansight.touches | getTimeline(args?), getContext(args?), getGestureSegments(args?), findTapTargets(args?), getHeatmap(args?), findDead(args?), getArtifacts(args?), summarizeFlow(args?) | Touch, gesture, heatmap, dead-touch, evidence, and flow contracts |
ansight.annotations | get(args?), create(args), update(args), delete(args) | AnnotationsResult, AnnotationMutationResult, AnnotationDeleteResult |
ansight.appTools | list() | AppToolListResult |
ansight.tasks | list(args?), run(args) | TaskDiscoveryResult, TaskCallResult<TOutput> |
ansight.lifecycle | launch(), foreground(), background(), terminate() | AppLifecycleResult |
ansight.device | setLocation(args), playLocation(args), clearLocation(), shake(args?), playAccelerometer(args), audioCapabilities(), injectMicrophoneAudio(args) | Device location and motion, AudioCapabilitiesResult, AudioInjectionResult |
ansight.permissions | names, grant(args), revoke(args), query(args) | PermissionResult with observed status and exact native mappings |
ansight.permissions.ios | names, grant(args), revoke(args), query(args), reset(args) | Native iOS service authorization on Simulator |
ansight.permissions.android | names, grant(args), revoke(args), query(args) | Native Android manifest permission authorization |
ansight.keyboard | open(args), isOpen(), dismiss(args?) | KeyboardActionResult, KeyboardStateResult |
ansight | callTool<TResult>(name, args?) | Structured result of an extra declared host tool |
For when to call each method, see Host API Methods. The generated declaration gives the exact argument and result shapes for your installed CLI version.
Native app permissions
Permission APIs operate on the task’s enforced application and native device in
SDK or device mode. No hostTools declaration is needed. Await calls serially:
import { Permission, IosPermission, AndroidPermission } from "./ansight-task.js";
await ansight.permissions.grant({ permission: Permission.Microphone });
const access = await ansight.permissions.query({ permission: Permission.Microphone });
expect(access.status, { id: "microphone-granted" }).toBe("granted");
await ansight.permissions.revoke({ permission: Permission.Microphone });
// Use these within tasks for the corresponding platform.
await ansight.permissions.ios.reset({ permission: IosPermission.Photos });
await ansight.permissions.android.grant({ permission: AndroidPermission.Camera });
Named constants are exported by the generated ansight-task.js runtime module
with readonly literal types in ansight-task.d.ts. Permission.Camera,
IosPermission.LocationAlways, and AndroidPermission.RecordAudio provide
completion and compile-time validation. Their matching type names can annotate
variables. Strings and the lowercase .names catalogs remain supported,
including app-defined Android identifiers.
The Permission API Reference documents every constant’s platform availability, exact native mapping, OS/tool restrictions, and links to the official Apple and Android documentation. Every constant also includes these details in its editor hover documentation.
PermissionResult reports supported, isSuccess, status, backend,
nativePermissions, and the enforced session/app/device identifiers. Status is
granted, denied, notDetermined, limited, unsupported, or unknown.
Mixed native states produce limited; missing or unreadable state is unknown.
Query returns unsupported results for unavailable resources. Unsupported
mutations and native command failures throw. A mutation may change some native
permissions before failing; its recorded result preserves individual states.
Successful commands report the subsequently observed state, which may be unknown
if the native store cannot be read.
| Shared name | iOS service | Android equivalent |
|---|---|---|
camera | Query via TCC; mutation only if listed by installed simctl | CAMERA |
microphone | microphone | RECORD_AUDIO |
contacts | contacts | READ_CONTACTS |
calendar | calendar | Declared READ_CALENDAR / WRITE_CALENDAR |
photos | photos | Declared READ_MEDIA_IMAGES / READ_MEDIA_VIDEO on API 33+ with target SDK 33+; otherwise READ_EXTERNAL_STORAGE |
location | location (when in use) | Declared ACCESS_COARSE_LOCATION / ACCESS_FINE_LOCATION |
locationAlways | location-always | ACCESS_BACKGROUND_LOCATION on API 29+; foreground location on older versions |
mediaLibrary | media-library | READ_MEDIA_AUDIO on API 33+ with target SDK 33+; otherwise READ_EXTERNAL_STORAGE |
motion | motion | ACTIVITY_RECOGNITION on API 29+ |
notifications | Unsupported by this simctl provider | POST_NOTIFICATIONS on API 33+ |
Android identifiers above have the android.permission. prefix. Mappings are
resource approximations: Android contacts are read access, calendar and location
use the permissions the app declares, photos cover declared visual media types,
and media library covers shared audio. nativePermissions exposes the exact
subset. On Android 14+, selected photo access is reported as limited and is also
revoked by shared revoke({ permission: "photos" }). Background location grants first grant declared
foreground dependencies; revocation removes background access only on API 29+. On iOS, revoking
location-always denies location access entirely, following simctl behavior.
Android supports emulators and authorized physical devices through ADB, uses the
active Android user, and requires declared permissions changeable through pm.
Special access and app account roles are outside this API. iOS supports Simulator
only. Native iOS service names also include photos-add, contacts-limited,
reminders, and siri. Mutations use the installed simctl privacy service list;
reset restores prompting on the next request. Queries read Simulator TCC and
location authorization stores without changing them. Unrecognized store schemas
return unknown. These describe app authorization, not global device switches or
the ability of a particular hardware resource to function in Simulator.
Permission changes can terminate the running application. Subsequent calls still
use the recorded session target; use ansight.lifecycle.launch() if the workflow
needs to relaunch it. Native grants can bypass usage-description checks, so keep
separate tests for the app’s actual permission request flow.
Execution capabilities
Use descriptor version 2 for tasks that support
external sessions. Version 1 tasks
retain their SDK requirement. Declare essential capabilities in requires so
the host can refuse an incompatible session before running task code:
import type { TaskDefinition, TaskInvocation } from "./ansight-task.d.ts";
export const task = {
schemaVersion: 2,
title: "Capture notes database",
description: "Save the simulator app's notes database to the session.",
requires: { capabilities: ["files.capture"] }
} satisfies TaskDefinition;
export default async function run({ ansight }: TaskInvocation) {
return await ansight.files.capture({
root: "data",
path: "Documents/notes.sqlite"
});
}
That example uses an iOS data-container path. An Android app might instead
store the file at files/notes.sqlite or databases/notes.sqlite.
await ansight.capabilities.snapshot() returns sessionId, executionMode,
live, appAvailable, executable appTools, and a capabilities map. Each
capability contains available, provider, and an unavailable reason.
Use await ansight.capabilities.require({ capabilities: ["files.read"] })
for an optional branch that needs extra support. Available keys include
ui.semantic, ui.screenshot, lifecycle, host.spans, evidence.read,
evidence.logs, evidence.telemetry, evidence.artifacts, logs.native,
files.read, files.capture, and app.tools; actual availability is session-specific.
app.available is false without a connected SDK provider. Calling any SDK
app.* operation then rejects with code: "capability_unavailable", the
missing capability, executionMode, and retryable: false. SDK-required
operations include setup guidance. For a task that needs a particular app
tool, declare requires: { appTools: ["files.list_directory"] } rather than
assuming the suite exists. Missing declared requirements refuse the task;
they do not skip the operation and report success.
External sandbox files
ansight.files uses the host’s external provider. SDK file tools remain under
app.files. Require files.read for list or read, and files.capture for
capture.
All three methods take a root-relative path and optional root (default
"data"). An empty path lists the root. iOS also supports "app" and
"group:<identifier>"; Android supports "data" only. Traversal and symlinks
are refused. Android access needs run-as or an already provisioned root ADB
daemon.
read returns base64 content and defaults to a maximum of 256 KiB; it cannot
exceed that limit. capture defaults to 16 MiB and accepts maximumBytes up to
64 MiB, returning a retained snapshotId. Both return size, SHA-256, provider,
timestamps, and consistency: "best-effort-live-copy". The automatic monitor’s
exit-copy limit is separately fixed at 16 MiB per file.
Host-observed spans
ansight.spans.measure(name, asyncOperation, { group }?) brackets an operation
and emits <name>.started, <name>.completed, or <name>.failed events. It
uses two actions and requires host.spans. Use these events as Trends anchors
for automation, or native log anchors
for flows the app already logs.
Location route playback
ansight.device.playLocation(args) starts a GPX (or KML) route on the task’s live
session device. Supply XML in routeContent, rather than a local path:
const playback = await ansight.device.playLocation({
sourceFileName: "approach.gpx",
routeContent: `<gpx version="1.1"><trk><trkseg>
<trkpt lat="-35.094" lon="150.804" />
<trkpt lat="-35.095" lon="150.805" />
</trkseg></trk></gpx>`,
mode: "fixed-speed",
fixedSpeedKph: 20,
});
// Continue interacting with the app while the device moves.
The promise resolves after the first location is set and rejects if playback
cannot start. The result includes runId, deviceId, platform, pointCount,
and the effective timing settings. The device must be a booted simulator or emulator.
The default mode: "recorded" uses track timestamps. speed scales recorded
timing from 0.1 to 100 (default 1). Without timestamps, playback falls back to
fixedSpeedKph (0.5–500, default 30); speed does not scale fixed speed.
loop: true repeats the route until stopped or replaced.
Playback shares the CLI’s single host-managed replay service. Starting a route
replaces any active route, and playback continues after the task ends. Coordinate
tasks that use location playback. Use ansight device location status to inspect
progress or later failures, and ansight device location stop to stop it.
clearLocation() clears the simulated coordinate but does not stop a running route.
Motion simulation
Use ansight.device.shake() to post a UIKit shake gesture to the task’s iOS
Simulator, or send alternating accelerometer pulses to its Android Emulator.
On Android, set intensity (5–50 m/s²), repetitions (1–8 pulse pairs), and
intervalMs (40–500 ms per pulse) when the default shake is unsuitable:
await ansight.device.shake({ intensity: 24, repetitions: 3, intervalMs: 90 });
// Assert the app's shake response separately.
On iOS Simulator, call shake() without options. The gesture is fixed and
received through UIKit’s responder chain. Ansight posts it to the selected
Simulator through CoreSimulator; it does not inject Core Motion sensor samples.
For a controlled motion trace, use playAccelerometer. Each value is held for
holdMs before the next sample. Values are in m/s², each axis is limited to
−100–100, each hold is 10–1000 ms, and a sequence can have 1–100 samples and
last at most 10 seconds:
await ansight.device.playAccelerometer({
samples: [
{ x: 0, y: 9.81, z: 0, holdMs: 100 },
{ x: 18, y: 0, z: 0, holdMs: 80 },
{ x: -18, y: 0, z: 0, holdMs: 80 },
],
});
Android calls finish after all samples have been sent and the emulator’s previous acceleration has been restored. The result reports the sample or gesture count, duration, target device, and delivery backend; it does not confirm that the app handled the motion. Android Emulator supports sensor injection through its console. Physical-device motion is not supported by these methods.
Microphone audio input
Complete the microphone setup checklist before running an audio task. iOS audio tasks require one selected simulator and serial execution of the entire recording workflow; the injection lock alone does not protect concurrent recordings. See audio concurrency restrictions.
ansight.device.audioCapabilities() inspects the task’s existing live session
without playing audio. Both audio APIs reject with
audio-parallel-execution-unsupported in parallel test batches (including
multi-device matrix batches); the error is logged in test progress and the task
call audit. Run audio tests serially on one device without --parallel.
On an allowed serial run, audioCapabilities() returns available, a diagnostic code and message,
provider details, the shared route scope, and accepted format/operation limits.
ansight.device.injectMicrophoneAudio({ file, timeoutMs?, waitForMicrophoneMs? }) delivers a
speech fixture to the session’s virtual microphone. The app must already be
recording through its normal flow and have microphone permission. This method
does not launch an app, press Record, or supply a file directly to a recognizer.
// The app in this task's pinned session is already recording.
const injection = await ansight.device.injectMicrophoneAudio({
file: "fixtures/audio/quote.wav",
timeoutMs: 30_000,
waitForMicrophoneMs: 0,
});
expect(injection.status, { id: "audio-delivery-completed" }).toBe("completed");
// Assert the app's transcript or saved recording separately.
fileis relative torun.repositoryRootPath. Absolute paths, traversal outside that root, and symlinks resolving outside it are rejected. The host validates and snapshots the exact bytes before delivery. This API rule does not make trusted repository TypeScript a filesystem sandbox.- Input is RIFF/WAVE PCM16, mono, 16 kHz, at most 15 seconds and 1 MiB. Compressed audio is not converted automatically.
timeoutMsdefaults to 30000, accepts 100–60000, and is capped by the task’s remaining deadline.waitForMicrophoneMsdefaults to provider preflight without waiting (0) and accepts 0–10000. Android checks guest microphone readiness; iOS at zero wait verifies its route only, so the app must already be recording. Unsupported or unhealthy readiness is reported explicitly.- Calls are serial and awaited. Injection counts as one task action and blocks through delivery and cleanup. Task cancellation/deadline cancellation stops the provider; partial or uncertain delivery is never retried automatically.
- Success returns
AudioInjectionResultwith fixture SHA-256 and format, submitted frames, delivery timing/provider diagnostics, and a retainedevidenceId.captureVerifiedandtranscriptionVerifiedarefalse: provider delivery does not establish what the app recorded or recognized. - iOS requires a separately installed loopback driver, exactly one booted simulator, and a verified input route in the configured Xcode’s running Simulator UI. The provider currently requires English audio-input menus and Accessibility access. It serializes injections across the shared host route. Readiness observes a host input client where supported, not the target app’s private recording stream. Android requires an authenticated, healthy emulator microphone provider. Emulator 37.1.11 has an observed native injection race that can crash the emulator; readiness guards do not guarantee stability. Ansight does not restart or retry automatically. Physical devices are unsupported.
The same operation is available through ansight audio inject.
These standard methods require no hostTools declaration and cannot target a
session, app, or device outside the task’s enforced session.
UI selectors
UiSelector is shared by find, wait, assert, and UI actions:
interface UiSelector {
nodeId?: string;
automationId?: string;
text?: string;
role?: string;
type?: string;
ancestorAutomationId?: string;
action?: string;
visible?: boolean;
enabled?: boolean;
exact?: boolean;
caseSensitive?: boolean;
index?: number;
}
Use ansight.ui.assert for evidence-producing expectations and waitFor for a
bounded state transition. UiAssertArguments additionally supports
expectedCount, expectedText, expectedValue, expectedVisible,
expectedEnabled, and an optional actionId.
tap, typeText, swipe, scroll, pinch, and back persist before-and-after
evidence and return UiActionResult. runSequence accepts from 1 through 32
already-known coordinate tap, swipe, pinch, or back actions and returns
per-step outcomes plus sequenceEndpoints evidence. Use separate actions when
the next step depends on an intermediate screen or when the evidence trail needs
a checkpoint around every input.
The includeScreenshot action option controls whether the underlying host result
contains final image content. Repository task methods receive structured action
and evidence data. Set includeScreenshot: false in a task when inline image
transport is unnecessary.
Software keyboard
ansight.keyboard.open(args) accepts the same stable selector fields as other
targeted UI actions. It focuses the selected text input and verifies the opened
keyboard through fresh device accessibility evidence. isOpen() performs a
fresh read without input. dismiss(args?) is idempotent: it sends the platform
back or Escape input only when device accessibility first confirms that the
keyboard is open, then verifies that it closed. This guard prevents a dismiss
call made against an already-closed keyboard from navigating away from the
current screen.
Keyboard state is intentionally unavailable when the host cannot capture a device accessibility hierarchy that reports system keyboard presence. The methods fail explicitly in that case instead of inferring visibility from a focused text field or returning an unverified success.
Time windows
Network, log, telemetry, touch, annotation, artifact, and timeline methods accept their
typed argument contracts from ansight-task.d.ts. Time-bound contracts use
ISO-8601 UTC timestamps and bounded result limits. Prefer typed methods over
ansight.callTool because they inject the task session and return stable named
contracts.
Captured network requests
ansight.network inspects HTTP capture records already received by the task’s
session. Its standard methods require no task.hostTools declaration. Calls
must be awaited serially, like other host methods.
const failures = await ansight.network.get({
methods: ["POST"],
query: "/checkout",
failedOnly: true,
limit: 20
});
const first = failures.requests[0];
if (first) {
const detail = await ansight.network.getRequest({ requestId: first.id });
const response = await ansight.network.readBody({
requestId: first.id,
side: "response",
maxBytes: 16 * 1024
});
// Header arrays preserve duplicate names; body content stays separately bounded.
console.log(detail.request.responseHeaders, detail.headersTruncated);
if (response.body !== null) {
console.log(response.body.encoding, response.body.data);
}
}
get accepts inclusive request-start startUtc/endUtc bounds, methods,
statuses, host, query, failedOnly, limit, and cursor. Different filters
combine with AND; values within methods or statuses combine with OR. Empty
arrays impose no restriction. Methods, hostname substrings, and text queries are
case-insensitive. Statuses accept integer codes from 100 through 599 or "1xx"
through "5xx". query searches URL, method, and error message. failedOnly: true
includes transport errors and HTTP statuses of at least 400; false imposes no
restriction. Invalid dates, reversed time ranges, and invalid filters reject the
call.
Pages contain complete summaries without header values or body content, ordered
newest first with a request-ID tie-breaker. The default limit is 200 and the
maximum is 1,000; the output budget can reduce the actual page size. Continue
with get({ cursor: page.nextCursor }) after checking nextCursor is non-null.
Cursors retain the initial matching snapshot, so live arrivals do not shift
subsequent pages.
The same filters may be repeated, but conflicting filters, a different session,
or an expired cursor reject. Snapshots expire five minutes after creation and
may be evicted earlier by the bounded host cache (32 snapshots, 16 Mi characters
of summary data). After expiry or eviction, start a new query without a cursor;
this also includes new arrivals. A matching snapshot that exceeds the cache
budget rejects with a request to narrow the filters. matchedRequestCount
describes the snapshot and isTruncated means matching records remain after
the page.
getRequest rejects unknown request IDs and returns metadata plus bounded
request/response header arrays. headersTruncated reports omitted complete
entries. readBody rejects unknown request IDs, but returns body: null when
the selected request has no retained body. A captured empty body instead has
data: "" and returnedByteCount: 0. Missing capture data does not explain why
content is unavailable.
maxBytes bounds decoded bytes, defaults to 16 KiB, and accepts 1 through 64 KiB.
Reads return a prefix that preserves UTF-8 boundaries; binary content uses Base64.
encoding identifies the representation, while capturedBytes, totalBytes,
and returnedByteCount distinguish capture metadata from this read. truncated
means capture omitted content; isTruncated means the read omitted retained
content. Both must be false for a complete body. Capture sanitization is preserved,
and body content is not automatically parsed as JSON. Metadata, headers, and
body previews are subject to the host’s serialized output budget; an individual
summary that cannot fit rejects explicitly.
Empty results do not prove capture is enabled or that an in-flight request has
completed. These methods inspect retained traffic; they do not initiate, replay,
intercept, or mock requests. Correlate network records with other evidence using
ansight.session.getTimeline({ categories: ["networkRequest"] }).
Timeline and screenshot annotations
An annotation has a timeline point or range and may include screenshot shapes
and a semantic visual-tree target. ansight.annotations.get(args?) keeps its
existing time, label, frameId, and hasGeometry filters. create, update,
and delete operate within the same enforced session and require no extra host
tool declarations.
const created = await ansight.annotations.create({
label: "Checkout failed",
startUtc: failedRequest.startedAtUtc,
endUtc: failedRequest.completedAtUtc,
notes: `${failedRequest.method} ${failedRequest.url}`,
geometries: [{
kind: "rectangle",
frameId: inspectedFrame.frameId,
x: 0.1, y: 0.65, width: 0.8, height: 0.12
}]
});
await ansight.annotations.update({
annotationId: created.annotationId,
expectedSource: created.annotation.source,
notes: "Response body reviewed.",
endUtc: null
});
// Remove the annotation if the task no longer needs it.
await ansight.annotations.delete({ annotationId: created.annotationId });
The example assumes the request and screenshot were inspected earlier in the
task; its coordinates are illustrative. Derive real shapes from the selected
frame. Annotation writes preserve evidence but do not establish task success;
use a named expect assertion after writing failure evidence.
create requires a non-whitespace label; it accepts optional annotationId,
notes, source, startUtc, endUtc, geometries, and target. It generates
an ID when omitted and atomically rejects an existing explicit ID. The default
source is task:<taskId>. An omitted start uses the session’s latest captured
timestamp; supply an evidence timestamp for reproducible placement. A generated
ID is not an idempotency key: reconcile an uncertain create before retrying.
update patches the latest saved record atomically. Omitted fields remain
unchanged. null clears notes, endUtc, or target; geometries: [] clears
shapes; a supplied geometry array replaces the entire array. Empty notes normalize
to no notes. Source, capture group, custom data, attached evidence, and hook
failures are preserved. The resulting end cannot precede its start, and equality
normalizes to a point. Timeline edits preserve actual screenshot capture times.
A no-op update succeeds and returns the saved record.
update and delete require an existing annotationId. Their optional
expectedSource is an exact-match guard checked atomically with the mutation;
a mismatch rejects without changing the record. This guard checks provenance,
not a revision. Concurrent patches preserve omitted fields; competing edits to
a supplied field use the last write. Mutations return the saved normalized
annotation, or deletedAnnotation after deletion. If the full saved or deleted
record exceeds 480,000 serialized characters, the call rejects before mutating
state. Existing host and CLI annotation operations remain available for large
legacy records.
Geometry inputs are typed point, rectangle, ellipse, or freeDraw records,
anchored by frameId. Coordinates are finite and normalized from 0 to 1 over
the full screenshot. Boxes need positive dimensions and must fit within the
image; free-draw paths need at least two points. Invalid geometry and styles
reject. The host resolves the frame’s capture timestamp and derives free-draw
bounds. Optional geometry fields are geometryId, text, strokeColor, and
strokeWidth. Stroke colors accept #RRGGBB or #AARRGGBB; stroke width must be
finite and positive.
A semantic target is { visualTreeSnapshotId, nodeId } from an inspected,
retained visual-tree snapshot. The host resolves its label, type, automation ID,
and bounds, and returns the node identity as target.targetId. Both frames and
trees must belong to the enforced session; references never silently capture
new evidence. Read results provide named geometry, target, inferred-shape, and
attached-evidence types. Output geometries also describe legacy line and arrow
shapes created by other host clients.
Task composition
ansight.tasks.list(args?) searches the same repository and App ID as the
calling task. It accepts query, feature, and maxResults; focused queries
use the same metadata, input-schema, synonym, and typo matching as
ansight task list.
Use the exact discovered ID to invoke another task against the same live session:
type ChildOutput = { selectedArea: string };
const catalog = await ansight.tasks.list({
query: "select map area",
feature: "map"
});
const selected = catalog.tasks.find(candidate => candidate.taskId === "map.select-area");
if (!selected) throw new Error("The map selection task is unavailable.");
const child = await ansight.tasks.run<ChildOutput>({
taskId: selected.taskId,
input: { area: "Secret Garden" }
});
expect(child.output?.selectedArea, { id: "area-selected" }).toBe("Secret Garden");
run resolves only when the child task is Passed; every other child status
rejects the call and therefore fails the caller unless its code deliberately
handles the error. The child keeps its own run ID, assertions, timeout, action
budget, tool-call audit, and persisted result. The parent records
ansight.tasks.run as one action, and its remaining deadline caps the complete
child call. Parent and child assertions are not merged, so the parent must still
record at least one named assertion for its own outcome.
The host reloads the caller’s exact workspace for discovery and execution, pins every child to the caller’s App ID and session, rejects self-calls and recursive cycles, and permits at most eight task execution levels in one call chain. A child cannot select another repository, app, session, or device.
app: live app API
Every standard app method returns Promise<AppToolCallResult<TResult>>. The
host pins the request to the task session, while the connected app enforces tool
availability, grants, argument schemas, and critical-operation policy.
| Namespace | Methods |
|---|---|
app.artifacts | query(args?), request(args) |
app.ui | getVisualTree(args?), getScreenshot(args?), inspectNode(args?), showOverlay(args?), getOverlay(args?), queryOverlays(args?), updateOverlay(args?), removeOverlay(args?), clearOverlays(args?) |
app.files | listDirectory(args?), readFile(args?), getChecksum(args?), download(args?), beginBinaryDownload(args?), push(args?), copy(args?), move(args?), delete(args?) |
app.fileDescriptors | listOpen(args?), countOpen(args?), inspect(args?), getUsage(args?) |
app.jniReferences | captureGraph(args?) |
app.preferences | listKeys(args?), get(args?), set(args?), remove(args?) |
app.clipboard | getText(), hasText(), setText(args), clear() |
app.secureStorage | get(args?), set(args?), remove(args?) |
app.data | listDatabases(args?), describeSchema(args?), query(args?) |
app.reflection | listRoots(args?), inspectObject(args?), describeType(args?), setMemberValue(args?), invokeMethod(args?) |
app.maui | getCurrentPage(args?), getVisualTree(args?), findElements(args?), getElement(args?), getBindableProperty(args?), setBindableProperty(args?), clearBindableProperty(args?), inflateXaml(args?), addElement(args?), removeElement(args?), setAppTheme(args?), getBindingContext(args?), getBindings(args?), getResourceState(args?), getNavigationState(args?), invokeElementAction(args?), waitForUi(args?), getLayoutDiagnostics(args?), getHandlerDiagnostics(args?), invokeBindingContextCommand(args?), setBindingContextProperty(args?) |
app.react | getComponentTree(args?), getShadowTree(args?), findComponents(args?), getComponent(args?), getNavigationState(args?), invokeComponentAction(args?) |
app.flutter | getWidgetTree(args?), inspectWidget(args?), findWidgets(args?), getNavigationState(args?) |
app.capacitor | getDocument(args?), inspectNode(args?), querySelector(args?), invokeAction(args?) |
app | callTool<TResult>(toolId, args?) for an app-defined tool |
app.clipboard uses the native Clipboard suite registered in the connected app.
Reads require Read policy; set and clear require Write policy. Text is limited to
65,536 UTF-8 bytes, and iOS can show its normal paste-permission prompt. See
Clipboard Tools for platform behavior and registration.
Standard suites require the corresponding SDK tool registration. A method can exist in the TypeScript contract while being unavailable in the selected app or denied by its runtime guard. The App Tool API explains when to use each method and how task calls differ from trigger action factories.
AppToolCallResult<TResult>
App results use a discriminated envelope:
type AppToolCallResult<TResult> =
| {
responseType: "tool.result";
sessionId: string;
appId: string;
toolId: string;
payload: {
success: true;
toolId: string;
result: TResult;
message?: string | null;
};
}
| {
responseType: "tool.error";
sessionId: string;
appId: string;
toolId: string;
payload: {
success?: false;
code: string;
message: string;
retryable: boolean;
details?: JsonValue | null;
};
};
The full envelope can also contain host-applied defaults, associated
artifacts, and persisted-visual-tree metadata. Narrow on responseType before
reading the typed success result.
expect: named assertions
| Matcher | Behavior |
|---|---|
expect(actual, { id, message? }).toBe(expected) | Uses Object.is, matching Playwright’s strict identity matcher. |
.toEqual(expected) | Uses strict recursive deep equality. |
.toBeTruthy() / .toBeFalsy() | Checks JavaScript truthiness. |
.toBeDefined() / .toBeUndefined() / .toBeNull() | Checks nullish states explicitly. |
.toContain(expected) | Checks string inclusion or direct array membership. |
.toContainEqual(expected) | Checks deep-equal array membership. |
.not.<matcher> | Negates the following matcher. |
expect.soft(actual, metadata).<matcher> | Records a failure and continues the task function. |
Assertion IDs must be stable and unique. At least one assertion must pass and
none may fail for the task status to be Passed. Hard failures stop the task
function; soft failures continue. A recorded failure remains authoritative even
if task code catches the hard assertion error. Returning normally without an
assertion produces Inconclusive.
Execution rules
- Await host and app calls serially; overlapping calls are rejected.
- Standard host APIs need no
hostToolsdeclaration. ansight.tasks.listandansight.tasks.runneed nohostToolsdeclaration.ansight.callToolaccepts only names declared intask.hostTools.app.callTooltargets app-defined tools and needs no descriptor declaration.- Every host, app, discovery, and child-task call counts toward the caller’s
maximumActions. - Tasks can access only the enforced live session through Ansight APIs.
- Repository TypeScript is trusted local code, not a filesystem or network sandbox.