Test Pencil input in Swift and SwiftUI
Add the optional Pencil package, bind your Swift and SwiftUI drawing surface, and play captured Simulator input with minimal app changes.
Requires Ansight CLI 1.6.0 or later and matching core SDK and Pencil packages at version 1.9.0 or later.
Dependencies and startup
Add the AnsightPencil SwiftPM product from the same SDK package as Ansight, or the AnsightPencil CocoaPod alongside the normal SDK. Use the repository URL with SwiftPM, or let CocoaPods resolve AnsightPencil, AnsightPencilRuntime and AnsightCore at the same version. The optional product depends on the existing core and its separate runtime; it adds no third-party swizzling library.
For an app with a Package.swift, replace <latest-sdk-version> with the latest SDK release tag version and add these entries to the existing dependency lists:
// Package.dependencies:
.package(url: "https://github.com/ansight-ai/ansight-sdk.git", exact: "<latest-sdk-version>")
// App target dependencies:
.product(name: "Ansight", package: "ansight-sdk"),
.product(name: "AnsightPencil", package: "ansight-sdk")
For an Xcode app target, select both products under its package dependencies. Rebuild the app after adding the product; no build-time source weaving or swizzling plugin is required.
At the app’s existing Ansight initialization point, on the main actor:
import Ansight
import AnsightPencil
let options = try AnsightOptions.createBuilder(.ansightDeveloperDefaults)
.withSimulatorPenInput()
.build()
try AnsightRuntime.shared.initializeAndActivateAnsightSdk(options: options)
The Pencil builder opt-in registers and authorizes only its exact playback tools on iOS Simulator. No ReadWrite setting or separate registration/arming call is needed. Existing grants for other tools stay unchanged; an explicit global tool disable still blocks Pencil, regardless of builder order. Capture settings are independent: touch/screenshot recording can be enabled for verification and replay, but is not required for playback.
Identify the existing native canvas
In the native drawing view’s setup, name the actual UIView and its existing hover recognizer:
canvas.accessibilityIdentifier = "drawing-canvas"
hoverRecognizer.name = "pencil-hover"
For hover, the recognizer must be enabled, idle, attached directly to canvas, and already accept Pencil input. If it is not configured for Pencil, its normal app setup needs:
hoverRecognizer.allowedTouchTypes = [NSNumber(value: UITouch.TouchType.pencil.rawValue)]
Keep the existing recognizer target/action and drawing callbacks. Do not add a second hover recognizer just to satisfy the test. Contact-only playback needs no hover recognizer.
Existing handlers read normal UIKit properties:
let pressure = touch.maximumPossibleForce > 0
? touch.force / touch.maximumPossibleForce : 0
let altitude = touch.altitudeAngle
let azimuth = touch.azimuthAngle(in: canvas)
if #available(iOS 17.5, *) {
let roll = touch.rollAngle
// Existing drawing logic uses roll.
}
For SwiftUI, configure the native view within the existing UIViewRepresentable implementation. Here DrawingCanvas is the app’s own UIKit drawing class:
struct CanvasHost: UIViewRepresentable {
func makeUIView(context: Context) -> DrawingCanvas {
let canvas = DrawingCanvas()
canvas.accessibilityIdentifier = "drawing-canvas"
// Name the existing hover recognizer in DrawingCanvas's own setup.
return canvas
}
func updateUIView(_ view: DrawingCanvas, context: Context) {}
}
A pure SwiftUI drawing view without an exposed native canvas/recognizer is not automatically adapted.
Optional recording
The startup example above adds playback without changing recording preferences. Keep existing capture configuration when integrating into an app. Pencil never enables recording as a side effect of playback.
Add these calls before .build() in the startup builder when retaining touch evidence and replay images:
.withTouchCapture()
.withSessionJpegCapture(intervalMilliseconds: 250)
Recording is independent of the permission granted by the Pencil opt-in.
Rebuilds and updates
Resolve the matching SwiftPM products or CocoaPods, then rebuild and relaunch the native app after adding or updating Pencil. No build-time source weaving or runtime package download is involved.
Changing the .pen.json file, its samples or host task arguments requires no application rebuild or package update. Startup options and surface-binding changes follow the app’s normal code lifecycle. See builds and package updates for the full cross-platform contract. Keep the CLI and optional SDK package compatible; protocol mismatches are rejected before input injection.
Connect, play, verify and finish
Start a matching CLI host before launching the Simulator app:
ansight host run
Build and launch the app on an iOS Simulator, open the drawing screen, then select its exact connected SDK session:
ansight session list --json
Confirm that the session belongs to this app and Simulator. For a recorded verification run, also confirm touch capture is active. Playback itself works with capture disabled and does not enable recording. A successful transport call alone is not proof that the app received the requested properties; test tasks must assert captured evidence and the app outcome.
Save this file as ansight/fixtures/drawing.pen.json in the app repository:
{
"schema": "ansight.pen-input/v1",
"samples": [
{ "offsetMs": 0, "phase": "down", "x": 0.3, "y": 0.5, "pressure": 0.2, "altitudeRadians": 0.5, "azimuthRadians": 0.7, "rollRadians": 0 },
{ "offsetMs": 150, "phase": "move", "x": 0.5, "y": 0.5, "pressure": 0.8, "altitudeRadians": 1.2, "azimuthRadians": 1.4, "rollRadians": 2 },
{ "offsetMs": 300, "phase": "up", "x": 0.5, "y": 0.5 }
]
}
Adjust coordinates so every point hits the selected surface. They are normalized to the app window in native mode, or the framework viewport in sdk mode, not to the target’s local bounds. Keep the drawing view stationary during playback.
From the repository root:
ansight ui play-pen --session <session-id> --file ansight/fixtures/drawing.pen.json --mode hid+sdk --target drawing-canvas
Inside an Ansight TypeScript task (this is the host automation SDK, not the application SDK):
const playback = await ansight.ui.playPen({
file: "ansight/fixtures/drawing.pen.json",
mode: "hid+sdk",
target: "drawing-canvas",
});
expect(playback.performed, { id: "pen-delivered" }).toBe(true);
await ansight.screenshots.take();
CLI paths resolve against the command’s working directory; task fixture paths resolve against the registered repository root.
For hover, extend the file using hoverEnter, hoverMove, and hoverExit with pressure zero and distance between 0 and 1. An up leaves proximity active: follow it with hoverMove or hoverExit, not a fresh hoverEnter. End contact with up or cancel. Files allow 2–2000 samples, strictly increasing offsetMs, at most 30 seconds and 1 MiB.
Verify three things separately:
- The command succeeds and the expected contact/hover samples appear in the retained session. Source labels distinguish simulated input.
- The app’s existing handler receives pressure, orientation and hover values; assert its drawing/model output, not only
performedor requested input. - Real input still works afterward. For native mode, a following HID-only control must have no override source label.
Open the retained session in the host’s player. Pencil-tip direction follows azimuth, length changes with tilt, the barrel highlight follows roll, hover lifts/fades, and contact shows pressure/tap feedback. Recording/rendering is independent of optional simulation registration.
When the run is complete, terminate the dedicated test app, retain its recording and confirm the exact session is disconnected:
ansight device terminate ios <simulator-udid> <bundle-id> --json
ansight session show <session-id> --json
Leave the host and Simulator running for replay. Do not terminate an unrelated/shared app instance.
For native hover, add --hover-recognizer pencil-hover to the CLI command, or hoverRecognizer: "pencil-hover" to the task call. UIKit hover properties require iOS 16.4 or later; roll requires iOS 17.5 or later.
Contact capture uses simulator-hid+sdk; hover uses simulator-sdk-hover.
Interception and compatibility
The host delivers native Pencil contacts through Simulator HID. The optional runtime correlates exact injected samples and supplies pressure/orientation through UIKit getter hooks. It drives only the named hover recognizer. Hooks are installed process-wide on first permitted hybrid playback, remain as pass-throughs after finish, and disappear at app exit. Conflicting getter implementations are checked before/during playback. This is not a guarantee for PencilKit, custom recognizers or other interception libraries; validate the app’s existing integration.
Lifecycle and failure behavior
Importing the package and building options do not install hooks. Normal SDK initialization registers the optional tools; there is no separate register() step. Unsupported platforms skip registration. Both Debug and Release Simulator builds are supported; physical-device and Mac Catalyst interception are excluded. There is no dependency on the consumer’s DEBUG symbol.
SDK framework playback uses ui.simulate_pen_framework and does not install UIKit hooks or inject HID. Native hybrid playback uses ui.simulate_pen. An active run expires after 35 seconds if abandoned. Explicit finish, errors and surface disposal release only the simulated pointer/hover owned by that run. Real input blocks or interrupts framework playback and continues to the app’s ordinary handlers.
If playback fails, check that the package is linked, initialization was awaited, the target is mounted and unique, tools are not globally disabled, the SDK is connected, and every point hits the target. Target movement, resizing, covering, unmounting or conflicting input stops playback. There is no silent fallback between backends.