Task UI Automation API

Use ansight.ui, ansight.keyboard, and ansight.screenshots for visible interaction and evidence on the task’s enforced session. See the host method index for every signature. Await calls serially and use a named expect to establish the task’s outcome.

Availability

Both platforms support virtual and physical UI targets when their providers are configured. An external physical iPhone uses Appium/XCUITest; external Android uses ADB/UIAutomator. See external monitoring prerequisites. Device input support is separate from SDK pairing or app-tool availability.

Device accessibility and SDK trees

ansight.ui.getLiveVisualTree() first attempts the device accessibility hierarchy. In SDK mode, it can fall back to MAUI, React Native, Flutter, Capacitor DOM, or native SDK trees. Supplying toolId or requesting includeProperties: true selects app-provided detail and requires SDK mode. In device mode, unavailable accessibility or app-only tree options reject.

The current declaration still types this method as an AppToolCallResult. The device-accessibility response instead has responseType: "device-accessibility" and payload.result, without the SDK success discriminator. Do not assume every response is "tool.result". Use the typed find, waitFor, and assert results for portable UI checks; inspect the actual envelope when consuming a live tree directly.

UI selectors

UiSelector is shared by find, wait, assert, and UI actions:

interface UiSelector {
  nodeId?: string;
  automationId?: string;
  text?: string;
  role?: string;
  type?: string;
  ancestorAutomationId?: string;
  action?: string;
  visible?: boolean;
  enabled?: boolean;
  exact?: boolean;
  caseSensitive?: boolean;
  index?: number;
}

Use ansight.ui.assert for evidence-producing expectations and waitFor for a bounded state transition. UiAssertArguments additionally supports expectedCount, expectedText, expectedValue, expectedVisible, expectedEnabled, and an optional actionId.

tap, typeText, swipe, scroll, pinch, and back persist before-and-after evidence and return UiActionResult. runSequence accepts from 1 through 32 already-known coordinate tap, swipe, pinch, or back actions and returns per-step outcomes plus sequenceEndpoints evidence. Use separate actions when the next step depends on an intermediate screen or when the evidence trail needs a checkpoint around every input.

The includeScreenshot action option controls whether the underlying host result contains final image content. Repository task methods receive structured action and evidence data. Set includeScreenshot: false in a task when inline image transport is unnecessary.

Software keyboard

ansight.keyboard.open(args) accepts the same stable selector fields as other targeted UI actions. It focuses the selected text input and verifies the opened keyboard through fresh device accessibility evidence. isOpen() performs a fresh read without input. dismiss(args?) is idempotent: it sends the platform back or Escape input only when device accessibility first confirms that the keyboard is open, then verifies that it closed. This guard prevents a dismiss call made against an already-closed keyboard from navigating away from the current screen.

Keyboard state is intentionally unavailable when the host cannot capture a device accessibility hierarchy that reports system keyboard presence. The methods fail explicitly in that case instead of inferring visibility from a focused text field or returning an unverified success.

Screenshots and assertions

Use ansight.screenshots.take(args?) for live capture, ansight.screenshots.getFrame(args?) for retained frames, and ansight.screenshots.assert(args?) for an SDK-mode comparison with a baseline. Keep a named assertion on the comparison result; taking a screenshot alone does not make a task pass. Exact options and result fields are in the task declaration.