Test Pencil input in MAUI and .NET iOS

Add the optional Pencil package, bind your MAUI and .NET iOS drawing surface, and play captured Simulator input with minimal app changes.

Requires Ansight CLI 1.6.0 or later and matching core SDK and Pencil packages at version 1.9.0 or later.

Dependencies

Keep the existing Ansight.Maui dependency and add the optional package in the app project. Use the same version for the core SDK and Pencil packages:

<ItemGroup>
  <PackageReference Include="Ansight.Pencil" Version="*" />
</ItemGroup>

Ansight.Pencil transitively includes Ansight.Pencil.Native.Apple.Binding and the separate precompiled AnsightPencilBridge.xcframework. The consumer links compiled native slices; it does not compile the shim’s Swift sources. Both Debug and Release Simulator slices include simulation. The consuming app’s DEBUG symbol does not change that binary.

The app’s existing AnsightRemoteToolsPolicy build setting must not be Disallowed. Its default AllowedWithWarnings permits tools with build warnings; Allowed explicitly permits them. This build policy is separate from runtime write authorization.

Configure playback

In MauiProgram.cs:

using Ansight.Pencil;

builder.UseAnsight<App>(options =>
{
    options.WithSimulatorPenInput();
});

The Pencil builder opt-in registers and authorizes only its exact playback tools on iOS Simulator. No ReadWrite setting or separate registration/arming call is needed. Existing grants for other tools stay unchanged; an explicit global tool disable still blocks Pencil, regardless of builder order. Capture settings are independent: touch/screenshot recording can be enabled for verification and replay, but is not required for playback.

Configure the existing iOS drawing handler

In the existing custom handler or native canvas constructor:

nativeCanvas.AccessibilityIdentifier = "drawing-canvas";
hoverRecognizer.Name = "pencil-hover";

Use the actual native UIView; do not assume every MAUI AutomationId maps to that view. The hover recognizer must already accept UITouchType.Stylus, be attached directly to that view, enabled and idle. Keep the existing callback. Contact-only playback needs neither a recognizer nor its name.

Normal C# UIKit reads remain unchanged:

var pressure = touch.MaximumPossibleForce > 0
    ? touch.Force / touch.MaximumPossibleForce : 0;
var altitude = touch.AltitudeAngle;
var azimuth = touch.GetAzimuthAngle(nativeCanvas);

Read RollAngle only on a supported OS (iOS 17.5+). A MAUI drawing library that hides its native view or hover recognizer requires integration with that library; the package does not replace its handlers.

Optional recording

The startup example above adds playback without changing recording preferences. Keep existing capture configuration when integrating into an app. Pencil never enables recording as a side effect of playback.

Add these calls inside the same UseAnsight configuration callback when retaining touch evidence and replay images:

options.WithTouchCapture();
options.WithSessionJpegCapture(new Ansight.SessionJpegCaptureOptions
{
    IntervalMilliseconds = 250
});

Recording is independent of the permission granted by the Pencil opt-in.

Rebuilds and updates

After adding or updating Pencil, restore compatible NuGet packages, rebuild and relaunch the app. The precompiled native framework changes only when the package/app is rebuilt; it is not downloaded or replaced during playback. The consumer does not need an Xcode build step for SDK Swift sources.

Changing the .pen.json file, its samples or host task arguments requires no application rebuild or package update. Startup options and surface-binding changes follow the app’s normal code lifecycle. See builds and package updates for the full cross-platform contract. Keep the CLI and optional SDK package compatible; protocol mismatches are rejected before input injection.

Connect, play, verify and finish

Start a matching CLI host before launching the Simulator app:

ansight host run

Build and launch the app on an iOS Simulator, open the drawing screen, then select its exact connected SDK session:

ansight session list --json

Confirm that the session belongs to this app and Simulator. For a recorded verification run, also confirm touch capture is active. Playback itself works with capture disabled and does not enable recording. A successful transport call alone is not proof that the app received the requested properties; test tasks must assert captured evidence and the app outcome.

Save this file as ansight/fixtures/drawing.pen.json in the app repository:

{
  "schema": "ansight.pen-input/v1",
  "samples": [
    { "offsetMs": 0, "phase": "down", "x": 0.3, "y": 0.5, "pressure": 0.2, "altitudeRadians": 0.5, "azimuthRadians": 0.7, "rollRadians": 0 },
    { "offsetMs": 150, "phase": "move", "x": 0.5, "y": 0.5, "pressure": 0.8, "altitudeRadians": 1.2, "azimuthRadians": 1.4, "rollRadians": 2 },
    { "offsetMs": 300, "phase": "up", "x": 0.5, "y": 0.5 }
  ]
}

Adjust coordinates so every point hits the selected surface. They are normalized to the app window in native mode, or the framework viewport in sdk mode, not to the target’s local bounds. Keep the drawing view stationary during playback.

From the repository root:

ansight ui play-pen --session <session-id> --file ansight/fixtures/drawing.pen.json --mode hid+sdk --target drawing-canvas

Inside an Ansight TypeScript task (this is the host automation SDK, not the application SDK):

const playback = await ansight.ui.playPen({
  file: "ansight/fixtures/drawing.pen.json",
  mode: "hid+sdk",
  target: "drawing-canvas",
});
expect(playback.performed, { id: "pen-delivered" }).toBe(true);
await ansight.screenshots.take();

CLI paths resolve against the command’s working directory; task fixture paths resolve against the registered repository root.

For hover, extend the file using hoverEnter, hoverMove, and hoverExit with pressure zero and distance between 0 and 1. An up leaves proximity active: follow it with hoverMove or hoverExit, not a fresh hoverEnter. End contact with up or cancel. Files allow 2–2000 samples, strictly increasing offsetMs, at most 30 seconds and 1 MiB.

Verify three things separately:

  1. The command succeeds and the expected contact/hover samples appear in the retained session. Source labels distinguish simulated input.
  2. The app’s existing handler receives pressure, orientation and hover values; assert its drawing/model output, not only performed or requested input.
  3. Real input still works afterward. For native mode, a following HID-only control must have no override source label.

Open the retained session in the host’s player. Pencil-tip direction follows azimuth, length changes with tilt, the barrel highlight follows roll, hover lifts/fades, and contact shows pressure/tap feedback. Recording/rendering is independent of optional simulation registration.

When the run is complete, terminate the dedicated test app, retain its recording and confirm the exact session is disconnected:

ansight device terminate ios <simulator-udid> <bundle-id> --json
ansight session show <session-id> --json

Leave the host and Simulator running for replay. Do not terminate an unrelated/shared app instance.

For hover, pass --hover-recognizer pencil-hover or hoverRecognizer: "pencil-hover". Captured sources are simulator-hid+sdk for contact and simulator-sdk-hover for hover.

The same process-wide getter-hook limitations as Swift apply.

Lifecycle and failure behavior

Importing the package and building options do not install hooks. Normal SDK initialization registers the optional tools; there is no separate register() step. Unsupported platforms skip registration. Both Debug and Release Simulator builds are supported; physical-device and Mac Catalyst interception are excluded. There is no dependency on the consumer’s DEBUG symbol.

SDK framework playback uses ui.simulate_pen_framework and does not install UIKit hooks or inject HID. Native hybrid playback uses ui.simulate_pen. An active run expires after 35 seconds if abandoned. Explicit finish, errors and surface disposal release only the simulated pointer/hover owned by that run. Real input blocks or interrupts framework playback and continues to the app’s ordinary handlers.

If playback fails, check that the package is linked, initialization was awaited, the target is mounted and unique, tools are not globally disabled, the SDK is connected, and every point hits the target. Target movement, resizing, covering, unmounting or conflicting input stops playback. There is no silent fallback between backends.