Test Pencil input in Flutter
Add the optional Pencil package, bind your Flutter drawing surface, and play captured Simulator input with minimal app changes.
Requires Ansight CLI 1.6.0 or later and matching core SDK and Pencil packages at version 1.9.0 or later.
Dependencies and startup
Requires Flutter 3.16+ and Dart 3.2+ for view-aware pointer routing. Add ansight_pencil alongside ansight_flutter using matching package versions, then run flutter pub get and rebuild the iOS app. Installing the Dart package or hot reload alone does not update native code.
Add the latest packages and let pub record their matching constraints:
flutter pub add ansight_flutter ansight_pencil
The native CocoaPods or SwiftPM dependencies resolve with the Flutter packages. Rebuild using your app’s existing iOS workflow.
In main.dart (keep your existing MyApp implementation):
import 'package:flutter/material.dart';
import 'package:ansight_flutter/ansight.dart';
import 'package:ansight_pencil/ansight_pencil.dart';
Future<void> main() async {
WidgetsFlutterBinding.ensureInitialized();
final options = createOptionsBuilder()
.withAnsightDefaults()
.withSimulatorPenInput()
.build();
await Ansight.instance.initializeAndActivate(options);
await AnsightFlutterInstrumentation.instance.install();
runApp(const MyApp());
}
The Pencil builder opt-in registers and authorizes only its exact playback tools on iOS Simulator. No ReadWrite setting or separate registration/arming call is needed. Existing grants for other tools stay unchanged; an explicit global tool disable still blocks Pencil, regardless of builder order. Capture settings are independent: touch/screenshot recording can be enabled for verification and replay, but is not required for playback.
Bind a Dart drawing widget
Wrap the existing drawing subtree. There is no native UIView identifier or hover recognizer to configure in SDK mode:
AnsightPencilSurface(
id: 'drawing-canvas',
child: Listener(
onPointerDown: handlePointerDown,
onPointerMove: handlePointerMove,
onPointerUp: handlePointerUp,
onPointerCancel: handlePointerCancel,
onPointerHover: handlePointerHover,
child: existingDrawingWidget,
),
)
The handler names and existingDrawingWidget above refer to the app’s existing drawing implementation. If it already contains a Listener, wrap that widget directly instead of adding another Listener. MouseRegion continues receiving stylus enter/exit through Flutter’s pointer routing.
The adapter sends a distinct simulated stylus pointer through Flutter’s GestureBinding, so normal hit testing and existing pointer handlers receive it. It rejects duplicate surface IDs, out-of-surface points, overlays, transformed wrappers, geometry changes and concurrent real contact. Unmounting removes the registration and releases the simulated pointer.
Inside your existing handler:
void handlePointerMove(PointerMoveEvent event) {
final pressure = event.pressure;
final tilt = event.tilt;
final orientation = event.orientation;
final roll = event.ansightPencil?['rollRadians'];
// Update the existing stroke model using these values.
}
Import package:flutter/gestures.dart where you use pointer event types. Pressure is normalized; tilt is measured from the surface normal. Orientation is the UIKit azimuth minus pi/2. Hover distance is normalized to 0–1. Flutter has no standard barrel-roll field: event.ansightPencil preserves source radians/distance on the original or transformed Listener event. This metadata is absent on ordinary real input. Do not require it in production handlers that also accept hardware events.
Optional recording
The startup example above adds playback without changing recording preferences. Keep existing capture configuration when integrating into an app. Pencil never enables recording as a side effect of playback.
Add these calls before .build() in the startup builder when retaining touch evidence and replay images:
.withTouchCapture()
.withSessionJpegCapture()
Recording is independent of the permission granted by the Pencil opt-in.
Rebuilds and updates
After adding or updating Pencil, run flutter pub get, resolve the native dependencies through the app’s CocoaPods/SwiftPM workflow, then rebuild and relaunch iOS. Flutter hot reload cannot add or update the native Pencil module.
Changing the .pen.json file, its samples or host task arguments requires no application rebuild or package update. Startup options and surface-binding changes follow the app’s normal code lifecycle. See builds and package updates for the full cross-platform contract. Keep the CLI and optional SDK package compatible; protocol mismatches are rejected before input injection.
Connect, play, verify and finish
Start a matching CLI host before launching the Simulator app:
ansight host run
Build and launch the app on an iOS Simulator, open the drawing screen, then select its exact connected SDK session:
ansight session list --json
Confirm that the session belongs to this app and Simulator. For a recorded verification run, also confirm touch capture is active. Playback itself works with capture disabled and does not enable recording. A successful transport call alone is not proof that the app received the requested properties; test tasks must assert captured evidence and the app outcome.
Save this file as ansight/fixtures/drawing.pen.json in the app repository:
{
"schema": "ansight.pen-input/v1",
"samples": [
{ "offsetMs": 0, "phase": "down", "x": 0.3, "y": 0.5, "pressure": 0.2, "altitudeRadians": 0.5, "azimuthRadians": 0.7, "rollRadians": 0 },
{ "offsetMs": 150, "phase": "move", "x": 0.5, "y": 0.5, "pressure": 0.8, "altitudeRadians": 1.2, "azimuthRadians": 1.4, "rollRadians": 2 },
{ "offsetMs": 300, "phase": "up", "x": 0.5, "y": 0.5 }
]
}
Adjust coordinates so every point hits the selected surface. They are normalized to the app window in native mode, or the framework viewport in sdk mode, not to the target’s local bounds. Keep the drawing view stationary during playback.
From the repository root:
ansight ui play-pen --session <session-id> --file ansight/fixtures/drawing.pen.json --mode sdk --target drawing-canvas
Inside an Ansight TypeScript task (this is the host automation SDK, not the application SDK):
const playback = await ansight.ui.playPen({
file: "ansight/fixtures/drawing.pen.json",
mode: "sdk",
target: "drawing-canvas",
});
expect(playback.performed, { id: "pen-delivered" }).toBe(true);
await ansight.screenshots.take();
CLI paths resolve against the command’s working directory; task fixture paths resolve against the registered repository root.
For hover, extend the file using hoverEnter, hoverMove, and hoverExit with pressure zero and distance between 0 and 1. An up leaves proximity active: follow it with hoverMove or hoverExit, not a fresh hoverEnter. End contact with up or cancel. Files allow 2–2000 samples, strictly increasing offsetMs, at most 30 seconds and 1 MiB.
Verify three things separately:
- The command succeeds and the expected contact/hover samples appear in the retained session. Source labels distinguish simulated input.
- The app’s existing handler receives pressure, orientation and hover values; assert its drawing/model output, not only
performedor requested input. - Real input still works afterward. For native mode, a following HID-only control must have no override source label.
Open the retained session in the host’s player. Pencil-tip direction follows azimuth, length changes with tilt, the barrel highlight follows roll, hover lifts/fades, and contact shows pressure/tap feedback. Recording/rendering is independent of optional simulation registration.
When the run is complete, terminate the dedicated test app, retain its recording and confirm the exact session is disconnected:
ansight device terminate ios <simulator-udid> <bundle-id> --json
ansight session show <session-id> --json
Leave the host and Simulator running for replay. Do not terminate an unrelated/shared app instance.
SDK capture uses simulator-sdk-framework. No UIKit swizzle or HID delivery occurs in this mode. This exercises Flutter’s event handling, not hardware timing or predicted/coalesced Pencil events. Native platform-view drawing should use native hybrid setup if its UIKit handlers are the behavior being tested.
Lifecycle and failure behavior
Importing the package and building options do not install hooks. Normal SDK initialization registers the optional tools; there is no separate register() step. Unsupported platforms skip registration. Both Debug and Release Simulator builds are supported; physical-device and Mac Catalyst interception are excluded. There is no dependency on the consumer’s DEBUG symbol.
SDK framework playback uses ui.simulate_pen_framework and does not install UIKit hooks or inject HID. Native hybrid playback uses ui.simulate_pen. An active run expires after 35 seconds if abandoned. Explicit finish, errors and surface disposal release only the simulated pointer/hover owned by that run. Real input blocks or interrupts framework playback and continues to the app’s ordinary handlers.
If playback fails, check that the package is linked, initialization was awaited, the target is mounted and unique, tools are not globally disabled, the SDK is connected, and every point hits the target. Target movement, resizing, covering, unmounting or conflicting input stops playback. There is no silent fallback between backends.