Test Pencil input in React Native

Add the optional Pencil package, bind your React Native drawing surface, and play captured Simulator input with minimal app changes.

Requires Ansight CLI 1.6.0 or later and matching core SDK and Pencil packages at version 1.9.0 or later.

Dependencies and startup

The optional package declares React Native 0.72+ and React 18+ peers and an iOS 15 minimum. Keep the React version required by your React Native release and use its higher iOS minimum where applicable. Add @ansight/react-native-pencil alongside the matching @ansight/react-native. The optional package brings @ansight/pencil-runtime as its JavaScript lifecycle helper and AnsightPencil through CocoaPods. Run the app’s normal pod install workflow and rebuild its native iOS binary. Expo needs a native development build containing these modules; JavaScript hot reload cannot link them.

Once the matching packages are available from your feed, install them with the same version:

npm install @ansight/react-native@latest @ansight/react-native-pencil@latest
cd ios
pod install

The shared runtime is a transitive dependency; consumers do not need to add it directly.

At the existing SDK startup point:

import Ansight, { createOptionsBuilder } from '@ansight/react-native';
import { AnsightPencil } from '@ansight/react-native-pencil';

const options = createOptionsBuilder()
  .withAnsightDefaults()
  .withPlugin(AnsightPencil)
  .build();
await Ansight.initializeAndActivate(options);

The Pencil builder opt-in registers and authorizes only its exact playback tools on iOS Simulator. No ReadWrite setting or separate registration/arming call is needed. Existing grants for other tools stay unchanged; an explicit global tool disable still blocks Pencil, regardless of builder order. Capture settings are independent: touch/screenshot recording can be enabled for verification and replay, but is not required for playback.

Bind the view and existing pointer handlers

Use the hook inside the existing drawing component. existingHandlers contains that component’s normal onPointerDown, onPointerMove, onPointerUp, onPointerCancel, and any hover enter/leave callbacks:

import React, { useRef } from 'react';
import { View, type ViewProps } from 'react-native';
import { usePencilSurface } from '@ansight/react-native-pencil';

export function DrawingSurface({ children, ...existingHandlers }: ViewProps) {
  const canvas = useRef<View>(null);
  const bindings = usePencilSurface('drawing-canvas', canvas, existingHandlers);
  return <View ref={canvas} {...bindings}>{children}</View>;
}

Spread the returned bindings onto the same View as the ref. They set testID to the surface ID and collapsable: false, preserve ordinary callbacks, and observe real touch start/end/cancel to avoid overlapping a real stroke. Do not override those bindings afterward. The native adapter resolves that exact native identifier and hit-tests each point; a Dart/DOM-style arbitrary identifier is not involved.

The SDK invokes the callbacks supplied to the hook with a synthetic React Native pointer-event shape. Existing pointer handlers read event.nativeEvent.pressure, tiltX, tiltY, twist, locationX/Y, pointerType: 'pen', and buttons. Angles tiltX/Y and twist are degrees. Optional nativeEvent.ansightPencil preserves exact source radians and hover distance; import AnsightPencilEvent if you need its TypeScript type.

Standard onTouchStart, onTouchMove, onTouchEnd and onTouchCancel handlers are also supported for contact; they receive force, touches and changedTouches. If both touch and pointer handlers are registered, both run, as their normal event subscriptions would. Hover uses pointer callbacks. Provide onPointerCancel or onTouchCancel for interrupted strokes and the handlers needed by the file. Missing handlers are rejected during preparation. IDs must be unique. Ref replacement, unmount, changed geometry, occlusion and concurrent real contact stop playback.

Keep the app’s existing React Native pointer-event configuration. The Pencil package does not enable global React Native feature flags. For example, the RN 0.86 verification app calls RCTSetDispatchW3CPointerEvents(true) from its Swift AppDelegate before creating the React Native factory, so ordinary native input reaches onPointerDown/Move/Up too. If your app uses standard touch callbacks instead, bind those callbacks. A successful synthetic callback test alone does not prove that the app enabled ordinary native pointer events.

Scope: SDK mode invokes only the registered view’s callbacks. It does not enter React Native’s native responder arbitration, synthesize Reanimated/Gesture Handler native events, or emulate bubbling to unregistered ancestors. A drawing library that consumes native gestures needs its own adapter or the existing UIKit hybrid path. Real events continue through the normal React Native system.

Optional recording

The startup example above adds playback without changing recording preferences. Keep existing capture configuration when integrating into an app. Pencil never enables recording as a side effect of playback.

Add these calls before .build() in the startup builder when retaining touch evidence and replay images:

.withTouchCapture()
.withSessionJpegCapture({ intervalMilliseconds: 250 })

Recording is independent of the permission granted by the Pencil opt-in.

Rebuilds and updates

After adding or updating Pencil, install matching npm packages, run pod install and rebuild/relaunch the native app. JavaScript reload cannot add or update the native module. Expo needs a rebuilt native development client containing these modules.

Changing the .pen.json file, its samples or host task arguments requires no application rebuild or package update. Startup options and surface-binding changes follow the app’s normal code lifecycle. See builds and package updates for the full cross-platform contract. Keep the CLI and optional SDK package compatible; protocol mismatches are rejected before input injection.

Connect, play, verify and finish

Start a matching CLI host before launching the Simulator app:

ansight host run

Build and launch the app on an iOS Simulator, open the drawing screen, then select its exact connected SDK session:

ansight session list --json

Confirm that the session belongs to this app and Simulator. For a recorded verification run, also confirm touch capture is active. Playback itself works with capture disabled and does not enable recording. A successful transport call alone is not proof that the app received the requested properties; test tasks must assert captured evidence and the app outcome.

Save this file as ansight/fixtures/drawing.pen.json in the app repository:

{
  "schema": "ansight.pen-input/v1",
  "samples": [
    { "offsetMs": 0, "phase": "down", "x": 0.3, "y": 0.5, "pressure": 0.2, "altitudeRadians": 0.5, "azimuthRadians": 0.7, "rollRadians": 0 },
    { "offsetMs": 150, "phase": "move", "x": 0.5, "y": 0.5, "pressure": 0.8, "altitudeRadians": 1.2, "azimuthRadians": 1.4, "rollRadians": 2 },
    { "offsetMs": 300, "phase": "up", "x": 0.5, "y": 0.5 }
  ]
}

Adjust coordinates so every point hits the selected surface. They are normalized to the app window in native mode, or the framework viewport in sdk mode, not to the target’s local bounds. Keep the drawing view stationary during playback.

From the repository root:

ansight ui play-pen --session <session-id> --file ansight/fixtures/drawing.pen.json --mode sdk --target drawing-canvas

Inside an Ansight TypeScript task (this is the host automation SDK, not the application SDK):

const playback = await ansight.ui.playPen({
  file: "ansight/fixtures/drawing.pen.json",
  mode: "sdk",
  target: "drawing-canvas",
});
expect(playback.performed, { id: "pen-delivered" }).toBe(true);
await ansight.screenshots.take();

CLI paths resolve against the command’s working directory; task fixture paths resolve against the registered repository root.

For hover, extend the file using hoverEnter, hoverMove, and hoverExit with pressure zero and distance between 0 and 1. An up leaves proximity active: follow it with hoverMove or hoverExit, not a fresh hoverEnter. End contact with up or cancel. Files allow 2–2000 samples, strictly increasing offsetMs, at most 30 seconds and 1 MiB.

Verify three things separately:

  1. The command succeeds and the expected contact/hover samples appear in the retained session. Source labels distinguish simulated input.
  2. The app’s existing handler receives pressure, orientation and hover values; assert its drawing/model output, not only performed or requested input.
  3. Real input still works afterward. For native mode, a following HID-only control must have no override source label.

Open the retained session in the host’s player. Pencil-tip direction follows azimuth, length changes with tilt, the barrel highlight follows roll, hover lifts/fades, and contact shows pressure/tap feedback. Recording/rendering is independent of optional simulation registration.

When the run is complete, terminate the dedicated test app, retain its recording and confirm the exact session is disconnected:

ansight device terminate ios <simulator-udid> <bundle-id> --json
ansight session show <session-id> --json

Leave the host and Simulator running for replay. Do not terminate an unrelated/shared app instance.

SDK capture uses simulator-sdk-framework. No HID or UIKit hooks are used. If the intended drawing logic lives in a native UIKit component instead of these JS pointer callbacks, use native setup and hid+sdk.

Lifecycle and failure behavior

Importing the package and building options do not install hooks. Normal SDK initialization registers the optional tools; there is no separate register() step. Unsupported platforms skip registration. Both Debug and Release Simulator builds are supported; physical-device and Mac Catalyst interception are excluded. There is no dependency on the consumer’s DEBUG symbol.

SDK framework playback uses ui.simulate_pen_framework and does not install UIKit hooks or inject HID. Native hybrid playback uses ui.simulate_pen. An active run expires after 35 seconds if abandoned. Explicit finish, errors and surface disposal release only the simulated pointer/hover owned by that run. Real input blocks or interrupts framework playback and continues to the app’s ordinary handlers.

If playback fails, check that the package is linked, initialization was awaited, the target is mounted and unique, tools are not globally disabled, the SDK is connected, and every point hits the target. Target movement, resizing, covering, unmounting or conflicting input stops playback. There is no silent fallback between backends.