Play repeatable Pencil input

Play pen files from the CLI and TypeScript tasks with native or framework delivery, explicit surface targeting and optional SDK capture.

Requires Ansight CLI 1.6.0 or later. SDK-assisted playback also requires matching core SDK and optional Pencil packages at version 1.9.0 or later.

Play the same UTF-8 .pen.json file from the CLI or the TypeScript task SDK:

ansight ui play-pen --session <session-id> --file ./demo.pen.json --mode hid+sdk --target drawing-canvas --hover-recognizer pencil-hover
const result = await ansight.ui.playPen({ file: "ansight/fixtures/demo.pen.json", mode: "hid+sdk", target: "drawing-canvas", hoverRecognizer: "pencil-hover" });
expect(result.performed, { id: "pen-delivered" }).toBe(true);

CLI paths resolve against the invoking working directory. Task paths resolve against the task repository root and cannot escape it, including through symlinks. The task’s session is pinned. Both calls await completion and release contact/proximity on cancellation. samples can be supplied instead of file for an in-memory sequence; supplying both is rejected.

{
  "schema": "ansight.pen-input/v1",
  "samples": [
    { "offsetMs": 0, "phase": "down", "x": 0.2, "y": 0.5, "pressure": 0.2, "altitudeRadians": 0.6, "azimuthRadians": 0.7, "rollRadians": 0 },
    { "offsetMs": 100, "phase": "move", "x": 0.5, "y": 0.5, "pressure": 0.9, "altitudeRadians": 1.2, "azimuthRadians": 1.4, "rollRadians": 3 },
    { "offsetMs": 200, "phase": "up", "x": 0.5, "y": 0.5 }
  ]
}

Files accept an optional description, 2–2000 samples, at most 1 MiB, and strictly increasing offsetMs from 0 through 30000. Coordinates are normalized to the current app window (hid/hid+sdk) or framework viewport (sdk), never to the target’s local bounds. Pressure and hover distance are 0–1. Angles are UIKit radians: altitude above the surface (0–pi/2), azimuth in window coordinates, and barrel roll. Defaults are zero pressure/distance/azimuth/roll and upright altitude. Orientations are not inferred from stroke direction.

Phases: down, move, up, cancel, hoverEnter, hoverMove, hoverExit. End every contact with up or cancel. up leaves the pen in proximity, so subsequent proximity samples use hoverMove; use hoverExit before a fresh hoverEnter. Non-contact samples require zero pressure. All samples are validated before input starts.

For a bound Flutter, React Native or Capacitor drawing surface, select framework playback:

ansight ui play-pen --session <session-id> --file ./demo.pen.json --mode sdk --target drawing-canvas
await ansight.ui.playPen({ file: "ansight/fixtures/demo.pen.json", mode: "sdk", target: "drawing-canvas" });

Install and bind the surface using the individual Swift, MAUI, Flutter, React Native or Capacitor guide.

Consumer setup is the optional package, builder opt-in and surface identifier/binding. Prepare, per-sample arming, HID/framework dispatch and cleanup belong to the host and Pencil package. The consumer does not implement those transport steps. Native dependency changes require an app rebuild; editing the pen file does not. No package is downloaded or replaced during playback. See build/update requirements and the dispatch sequence.

Backends and evidence

  • hid (default): iOS Simulator only; sends native Pencil contacts. Earlier CoreSimulator versions use SimulatorKit/Indigo; Xcode 27-era CoreSimulator uses an included helper inside the target simulator. Selection follows the shared CoreSimulator version, including when an older Xcode is selected; see simulator dependencies. HID-only contacts can reach UIKit with zero force. The Indigo serializer also reports default altitude/azimuth and loses fractional roll precision. Use hid+sdk when testing specified pressure and orientation values. These are private transport APIs with runtime-dependent fidelity.

  • hid+sdk: requires an iOS Simulator app that explicitly opts into experimental pen input, with the optional Pencil package and executable write-policy tool ui.simulate_pen. Swift apps add the separate AnsightPencil product and use .withSimulatorPenInput() in their options builder; MAUI apps add Ansight.Pencil, import its namespace and call options.WithSimulatorPenInput() in their UseAnsight configuration. Neither aggregate SDK includes this package. Both Debug and Release Simulator builds support this, including the separate precompiled Pencil framework in NuGet. HID still supplies real .pencil contacts. The optional package matches transport-stamped Pencil samples by timestamp, type, phase, exact view/window and position and temporarily overrides force, maximum force, altitude, azimuth/unit vector and roll getters. It requires a unique native target accessibility identifier and sends hover only to the explicitly named, idle Pencil hover recognizer directly on that target. This tests app behavior, not hardware fidelity.

  • sdk: requires ui.simulate_pen_framework, a mounted target bound by the optional Flutter, React Native or Capacitor package, and a connected SDK session. Recording is optional and is never enabled as a playback side effect. The host paces the validated samples through the framework adapter. Flutter routes stylus events through its pointer pipeline; React Native invokes the view’s registered pointer/touch callbacks; Capacitor dispatches DOM PointerEvents on the registered element. No HID or UIKit hooks are used. hoverRecognizer is not accepted. React Native native gesture arbitration and trusted DOM input are not emulated.

Native interception is excluded from physical-device builds. The core SDK contains no simulation implementation. Without the optional package and builder opt-in, the app does not advertise the tool or install any pen hooks. The opt-in grants the exact Pencil tools automatically; consumers make no separate permission or arming call. An explicit global tool disable still wins. Hybrid mode fails if the tool is absent or denied; it does not silently downgrade.

Flutter, React Native and Capacitor use the separate ansight_pencil, @ansight/react-native-pencil and @ansight/capacitor-pencil packages. Configure .withSimulatorPenInput() in Flutter or .withPlugin(AnsightPencil) in React Native and Capacitor, then await the normal SDK initialization call. The Pencil opt-in grants only its exact playback tools, including under a read-only guard. No broad write permission or capture setting is required. An explicit global tool disable still blocks playback. Enable capture separately when a recording is wanted, then assert on the recorded input and the app’s response. Native capture does not prove that a framework forwards every field to its Dart, React Native or DOM event handlers.

On the first permitted hybrid playback, the implementation swizzles UIKit getters throughout that app process. It supplies values only for matching Pencil samples during the run; finger and unmatched input call the original methods. Simulation state clears at completion and expires after 35 seconds if the host disappears. The hooks remain installed but pass through after completion, until the app exits. Competing getter implementations are checked before installation and during playback; detected conflicts stop playback without replacing the other implementation. Hover changes only the explicitly selected recognizer, refuses busy recognizers and cleans up only its own sequence. Target geometry, window or hit-test changes stop playback before the next injected sample. Queued HID delivery, coalescing, custom recognizers, PencilKit internals and app-specific event interception may affect fidelity. Validate the existing pen integration before relying on hybrid playback; opt-in is not a guarantee of compatibility.

performed records a successful transport operation; it does not prove that the app received input or rendered a result. Assert on captured touches and the app’s outcome. Hybrid contact samples carry details.inputSource = "simulator-hid+sdk"; hover uses "simulator-sdk-hover". Framework samples use "simulator-sdk-framework". Input without SDK overrides, including HID-only playback, leaves this field absent. Replay preserves these fields and draws a pencil tip with azimuth direction, tilt-dependent length, barrel-roll highlight, hover lift/fade, pressure and tap feedback.

Framework adapters reject duplicate/unmounted targets, changed geometry, covered points, concurrent real contact and mismatched sequences. Errors, disposal and the 35-second lease cancel only their own synthetic pointer. Test existing handlers and ordinary input after playback; app-specific drawing libraries may require their own adapter.