Simulate Apple Pencil input

Integrate optional Pencil playback on iOS Simulator, understand event delivery and isolation, and update packages without changing drawing handlers.

Pencil simulation is implemented separately from normal input recording. The aggregate SDKs do not depend on or activate it. All packages below support the optional native implementation on iOS Simulator. Flutter, React Native and Capacitor also provide explicit framework surface adapters.

App frameworkOptional packageOpt-in in the options builder
SwiftAnsightPencil SwiftPM product / CocoaPod.withSimulatorPenInput()
.NET / MAUIAnsight.Pencil NuGetoptions.WithSimulatorPenInput() during configuration
Flutteransight_pencil.withSimulatorPenInput()
React Native@ansight/react-native-pencil.withPlugin(AnsightPencil)
Capacitor@ansight/capacitor-pencil.withPlugin(AnsightPencil)

End-to-end guides by technology

  • Swift and SwiftUI: native view/recognizer setup, HID hybrid playback and compatibility.
  • MAUI and .NET iOS: NuGet/precompiled framework, handler setup and capture.
  • Flutter: AnsightPencilSurface, existing Dart pointer handlers and capture.
  • React Native: usePencilSurface, existing JS pointer callbacks and native hit testing.
  • Capacitor: bindSurface, existing DOM PointerEvent handlers and cleanup.

Each guide covers dependencies, startup, surface binding, capture, a pen-file example, CLI/TypeScript playback, checking input delivery and teardown. Use CLI 1.6.0 or later and matching SDK and Pencil packages at version 1.9.0 or later. Platform requirements and compatibility limits are described in each integration guide.

Two SDK delivery paths are explicit: hid+sdk targets a native UIKit canvas; sdk targets a registered framework surface without HID or UIKit hooks. sdk captures use simulator-sdk-framework. Framework adapters register ui.simulate_pen_framework; they require a connected SDK session regardless of whether touch capture is enabled. Their application-level events do not claim native gesture or hardware fidelity. See each guide’s scope and event mapping.

One opt-in; independent recording

The Pencil builder opt-in registers and authorizes only its exact playback tools on iOS Simulator. No ReadWrite setting or separate registration/arming call is needed. Existing grants for other tools stay unchanged; an explicit global tool disable still blocks Pencil, regardless of builder order. Capture settings are independent: touch/screenshot recording can be enabled for verification and replay, but is not required for playback.

The default unconfigured core guard grants no tools. A package opt-in can authorize its exact tools from that default; it does not turn on unrelated reads or writes. readOnly continues allowing ordinary read tools plus the opted-in Pencil tools. disabled is an explicit global prohibition and always wins. Grants are bounded by tool ID and maximum policy, not wildcards or the entire ui category. The core implements this generic permission mechanism without depending on Pencil.

The opt-in adds permissions; it does not remove permissions already granted by the app. Convenience presets such as Swift’s ansightDeveloperDefaults and MAUI’s UseAnsight may already allow other write tools. Preserve or choose the app’s global policy separately.

The host handles prepare, per-sample arming, HID dispatch and cleanup internally. Consumers only add the optional package, use the builder opt-in and identify/bind the drawing surface, then rebuild the app once. Pen-file changes need no rebuild. SDK capture/replay requires recording to be enabled; playback does not silently enable it.

Consumer changes at a glance

Add the optional package alongside the existing Ansight SDK, opt in at the existing SDK initialization point, and expose the intended drawing surface. Application variables such as canvas, hoverRecognizer and the drawing callbacks in the guides refer to the consumer’s existing implementation.

TechnologySurface setup in the consumerWhat existing code receivesPlayback mode
Swift / SwiftUI native canvasSet the actual UIView’s accessibilityIdentifier; name its existing hover recognizer only when testing hover.UIKit touch properties and the selected recognizer’s existing callbacks.hid+sdk
MAUI / .NET iOS native canvasSet AccessibilityIdentifier on the actual native UIView; set Name on its existing hover recognizer when needed.Normal C# UIKit touch properties and recognizer callbacks.hid+sdk
FlutterWrap the existing drawing widget in AnsightPencilSurface(id: ..., child: ...).Flutter stylus pointer events through normal hit testing. Roll is optional ansightPencil metadata.sdk
React NativeBind the same View ref and existing handlers with usePencilSurface, then spread its returned bindings onto that View.The bound pointer/touch callbacks; native Gesture Handler/Reanimated arbitration is outside this adapter.sdk
CapacitorBind the mounted DOM element with AnsightPencil.bindSurface; await its disposer before removing/replacing the element.Synthetic DOM PointerEvents with isTrusted: false.sdk

Native apps keep their existing drawing callbacks. Framework apps need the surface adapter shown above; enabling the startup plugin alone does not bind every drawing library automatically. Do not add a second hover recognizer or replace existing drawing handlers merely for playback. A native drawing component inside a framework app can use the UIKit hybrid path instead of its framework adapter.

Builds and package updates

All simulation paths are iOS Simulator only. Debug and Release Simulator binaries are supported. Physical iOS devices, Android, macOS and Mac Catalyst do not register playback tools; ordinary pen capture remains a separate core SDK feature. Native interception is excluded at compilation with the Simulator platform condition, not the consumer’s DEBUG symbol.

There is no runtime package download, binary patching, build-time source weaving or automatic SDK replacement. UIKit getter hooks are installed at runtime inside an opted-in Simulator process when native hybrid playback first prepares a run. Framework sdk playback does not install those hooks.

ChangeConsumer action
Add or update Swift Pencil products/podsResolve matching dependencies and rebuild/relaunch the native app.
Add or update MAUI Pencil NuGet packagesRestore the matching package graph, then rebuild/relaunch. NuGet supplies the precompiled framework; the app does not compile its Swift source.
Add or update Flutter PencilRun flutter pub get, resolve native dependencies using the app’s CocoaPods/SwiftPM workflow, then rebuild/relaunch iOS. Hot reload alone cannot add/update the native module.
Add or update React Native PencilInstall matching npm packages, run pod install, then rebuild/relaunch the native app. Expo requires a native development build containing the modules.
Add or update Capacitor PencilInstall matching npm packages, run npx cap sync ios, then rebuild/relaunch the native app.
Change .pen.json samples or host task argumentsPlay the updated file/task; no application rebuild or package update is needed.
Change app startup options or surface bindingsApply the normal app code build/reload lifecycle and rerun initialization/mounting as appropriate; pen playback does not change app configuration.

Keep the CLI, core SDK and optional packages from compatible builds. The host checks the playback protocol before injecting input and rejects a mismatch. When updating MAUI, restore the new NuGet versions and rebuild so the app links the matching precompiled frameworks.

Swift

Add the AnsightPencil product or CocoaPod alongside the SDK:

import Ansight
import AnsightPencil

let options = try AnsightOptions.createBuilder(.ansightDeveloperDefaults)
    .withSimulatorPenInput()
    .build()
try AnsightRuntime.shared.initializeAndActivateAnsightSdk(options: options)

withSimulatorPenInput() is supplied by the optional package. Building options only records the opt-in. SDK initialization registers the tool before returning; unsupported platforms register nothing. Repeated opt-in in the chain is deduplicated. AnsightPencil.register() and registerPencilSimulation() remain lower-level APIs for manual integration. Importing the module does not register tools or install hooks.

.NET / MAUI

Add Ansight.Pencil, then configure:

using Ansight.Pencil;

builder.UseAnsight<App>(options =>
{
    options.WithSimulatorPenInput();
});

The optional package owns a separate precompiled framework and binding. Debug and Release Simulator slices contain simulation; physical-device and Mac Catalyst slices exclude interception. The app’s DEBUG symbol does not control the framework. See the MAUI integration guide.

Flutter, React Native and Capacitor

Add the matching optional package and rebuild the native app (run CocoaPods installation or Capacitor sync as appropriate). Configure Pencil in the normal SDK builder chain. Flutter exposes .withSimulatorPenInput() as a Dart extension; React Native and Capacitor use .withPlugin(AnsightPencil). Await the usual SDK initialization call. No separate registration call is required.

The SDK runs and awaits optional setup after the native runtime is initialized. Building or copying options is passive. Repeated builder opt-in is deduplicated by plugin ID, initialization/linking failures propagate, and unsupported platforms skip Pencil registration. The core SDK knows only the generic plugin contract; it does not depend on Pencil. Registration does not install hooks; the first permitted native hybrid playback does.

See installation and startup examples for Flutter, React Native, and Capacitor.

These packages retain the native UIKit path and also expose explicit framework surface adapters in the per-technology guides. Use sdk to deliver framework events; use hid+sdk for a native drawing integration. Framework fields and gesture behavior differ, so validate the app’s own handler output, not just capture.

Playback and capture labels

Play the same .pen.json file from the CLI or an Ansight TypeScript task. These are host commands, not consumer app calls:

# Native canvas, with hover:
ansight ui play-pen --session <id> --file demo.pen.json --mode hid+sdk --target drawing-canvas --hover-recognizer pencil-hover

# Flutter, React Native or Capacitor framework surface:
ansight ui play-pen --session <id> --file demo.pen.json --mode sdk --target drawing-canvas
// Native canvas; omit hoverRecognizer for a contact-only file.
await ansight.ui.playPen({
  file: "ansight/fixtures/demo.pen.json",
  mode: "hid+sdk",
  target: "drawing-canvas",
  hoverRecognizer: "pencil-hover",
});

// Framework surface; sdk mode does not accept hoverRecognizer.
await ansight.ui.playPen({
  file: "ansight/fixtures/demo.pen.json",
  mode: "sdk",
  target: "drawing-canvas",
});

The calls above demonstrate alternatives; choose the mode that matches the app’s surface. The per-technology guides contain a complete input-file example, coordinate rules and capture assertions. The CLI pen-file reference documents validation and limits. File coordinates are normalized to the native window or framework viewport, not to the drawing surface’s local bounds.

Recorded input exposes details.inputSource so assertions can distinguish simulated values: simulator-hid+sdk for injected native contact properties, simulator-sdk-hover for synthetic native hover, and simulator-sdk-framework for framework playback. Input without SDK overrides has no source label.

Enable touch capture and screenshots separately when a replay is wanted. The guides show the extra builder settings; the minimal playback setup leaves existing/default recording preferences in place. A successful playback call or captured sample does not prove that a drawing library rendered a continuous stroke: also assert the app’s own handler/model output and check ordinary input afterward.

Who arms and dispatches input

For native hid+sdk playback:

  1. The host validates the file and sends prepare with the run ID, native target, samples and optional hover-recognizer name.
  2. The app-side Pencil runtime checks the target and hook ownership, then prepares the scoped run.
  3. Before each contact, the host stamps the HID sample and awaits the app’s arm acknowledgement. The same timestamp is carried in the subsequent Simulator HID dispatch.
  4. UIKit delivers the Pencil contact through its normal app event path. Getter hooks supply requested properties only when the touch matches the armed timestamp, type, phase, view, window and position. The consumer’s existing handlers read those properties normally.
  5. Hover is delivered in-process to the explicitly selected existing recognizer; it is not evidence of a hardware hover event from iOS.
  6. The host finishes the run and releases contact/proximity on cancellation. Run state expires if abandoned; getter hooks remain pass-throughs until process exit.

For framework sdk playback, the host sends prepare, timed sample commands and finish to the registered adapter. The adapter validates the bound surface, delivers framework events/callbacks, and records them only when capture is enabled. There is no HID dispatch or UIKit getter interception on this path. Neither path needs a consumer-written arm/match/forward implementation.

Native hybrid scope and compatibility

Hybrid playback requires target, the unique native accessibility identifier of the drawing view. All sample coordinates must hit that view or its descendants. The run pins the exact view and window; covering, moving, resizing, removing or changing the active window stops playback before the next sample. The target is selected per playback, so the options builder stays app-wide.

For hover, also provide hoverRecognizer, the name of one enabled, idle Pencil hover recognizer directly attached to that view. For example, set canvas.accessibilityIdentifier = "drawing-canvas" and hover.name = "pencil-hover" in Swift (AccessibilityIdentifier and Name in MAUI). Contact-only files need no hover recognizer. Playback never searches ancestors or drives other recognizers. Cleanup ends only the hover sequence it owns; it does not cancel a pre-existing gesture. The SDK reads the selected recognizer for recording without driving its observer recognizer. Normal hover-exit values remain available through the app’s terminal callback, and stop overriding properties once the recognizer resets.

The host stamps each injected contact with a monotonic HID timestamp aligned to whole nanoseconds to survive Indigo conversion and arms that sample before delivery. The SDK requires that timestamp (within one nanosecond for floating-point conversion), Pencil type, phase, window, target view and location to match. Cached values belong to a specific touch object and timestamp. There is no fallback to position-only matching. The host requires native playback protocol version 2 before injecting input; update both the CLI and optional package together. Timestamps are transport correlation, not an unforgeable UIKit event identity; unsupported Simulator timestamp behavior must be treated as a fidelity failure.

For WebKit targets, a continuing Pencil contact remains scoped to its original view even when WebKit clears the touch’s view. Recorded coalesced samples retain the pressure and simulation-source metadata associated with their transport timestamp. This requires compatible core SDK and optional Pencil packages in the app as well as the CLI.

For a native WKWebView, assign the native view a unique accessibility identifier and use it as the hid+sdk target. Verify the DOM callbacks separately from the SDK touch stream: trusted pen input and pointer capture can work while WebKit exposes no DOM coalesced samples or barrel twist. SDK-recorded roll and coalesced samples do not prove that those fields reach JavaScript. The native hover recognizer path also does not establish DOM hover behavior.

The first permitted playback still installs process-wide getter hooks. Before installation, every selected getter must resolve to UIKit; later playback checks that the installed hooks have not changed. A detected competing implementation fails with pen_hook_conflict without replacing the other library’s method. Finished runs clear their values, leaving pass-through hooks until app exit. This detects direct getter replacement, not every possible event interception technique. PencilKit, custom recognizer subclasses and framework event translation still require app-specific compatibility testing.

When using native hid+sdk mode, Swift, MAUI, Flutter, React Native and Capacitor share these restrictions. A framework identifier must reach the native view’s accessibility identifier; Dart widget keys and DOM element IDs alone are not native targets. Hover requires a named native recognizer, so integrations without one need native setup before opting into hover playback. The package does not discover or mutate unrelated framework recognizers automatically.

Package composition

Pencil stays outside the aggregate SDK dependency graph. Swift’s optional AnsightPencil product uses AnsightCore and its separate AnsightPencilRuntime. MAUI’s optional Ansight.Pencil brings a separate Ansight.Pencil.Native.Apple.Binding with the precompiled AnsightPencilBridge.xcframework; it does not embed a second core runtime. Flutter, React Native and Capacitor bring the native Pencil module through their normal native dependency tooling.

React Native and Capacitor also depend transitively on @ansight/pencil-runtime. Applications do not initialize that helper directly. It has no external runtime dependencies and owns shared validation, exclusive run ownership, cancellation, pointer mapping and the 35-second lease. Platform adapters own actual event delivery and capture. No third-party swizzling library is required.