VisualTree

Install

dotnet add package Ansight.Tools.VisualTree --prerelease

Register the Suite

using Ansight;
using Ansight.Tools.VisualTree;

var options = Options.CreateBuilder()
    .WithVisualTreeTools()
    .WithReadOnlyToolAccess()
    .Build();

Registration API

• WithVisualTreeTools(): registers the full visual tree, screenshot, and diagnostic overlay suite. No suite-specific configuration is currently required.
• WithReadOnlyToolAccess(): exposes visual tree, screenshot, node inspection, and read-only overlay tools.
• WithReadWriteToolAccess(): also exposes overlay mutation tools for drawing, updating, and clearing diagnostic overlays.

Visual-Tree Providers

Visual-tree capture is provider-based. VisualTreeProviderRegistry always exposes the platform hierarchy as native, and Ansight.Tools.Maui registers a separate maui source.

Additional frameworks can implement IVisualTreeProvider and register a named source:

using Ansight.Tools.VisualTree;

using var registration = VisualTreeProviderRegistry.Register(
    new CustomVisualTreeProvider());

The provider supplies a stable Source, a display name, GetVisualTreeAsync(...), and InspectNodeAsync(...). Pass source to ui.get_visual_tree or ui.inspect_node; omitting it selects native. VisualTreeProviderRegistry.Query() returns every registered source. Provider responses use the shared compact visual-tree v2 contract.

Local features can reuse the registry without enabling remote tools. For example, Annotated Feedback captures every registered source.

Specific Concerns

• The tools inspect the foreground scene only.
• Screenshot capture exports rendered UI content, so treat it as sensitive.
• ui.get_screenshot requires a live binary transfer channel from the paired host. The result returns transfer metadata, not inline base64.
• includeComputedStyles and detailed node inspection can reveal UI labels and accessibility metadata.
• Overlay tools draw input-transparent diagnostic rectangles over the active native window.
• Overlay mutation tools require write access. Persistent overlays created with durationMs = 0 stay visible until removed or cleared.

Important: Calling screenshot tools will result in an FPS drop while the current frame is captured, encoded, and transferred. Avoid screenshot-heavy investigations during trend measurements unless visual evidence is required.

Tool Matrix

Get Visual Tree

Arguments:

• includeBounds: include node bounds, default true
• includeComputedStyles: include implementation-specific properties, default false
• maxDepth: child depth limit, default 8
• rootNodeId: optional subtree root
• source: optional provider source; defaults to native
• maxNodes: maximum number of nodes to capture

Returns:

• platform
• source
• capturedAtUtc
• format
• types: capture-local type-name registry referenced by node typeId
• root

Each compact-v2 node includes semantic role, supported actions, interactability, stable automation id when available, platform-neutral presentation data, and structural child order. See the Visual-Tree Payload Contract before consuming the JSON directly.

Example:

{
  "toolId": "ui.get_visual_tree",
  "arguments": {
    "includeBounds": true,
    "includeComputedStyles": false,
    "maxDepth": 6
  }
}

Get Screenshot

For the full comparison between this on-demand tool and periodic session JPEG capture, see JPEG Screenshots.

Arguments:

• format: png or jpeg
• quality: JPEG quality 1-100
• maxWidth: optional width cap
• annotateNodeIds: overlay node ids on the screenshot
• afterScreenUpdates: Apple platforms wait for pending screen updates before rendering; default true

Returns:

• platform
• capturedAtUtc
• format
• width
• height
• deliveryMode = websocket_binary
• wireProtocol = ansight.file-transfer.v1
• transferId
• downloadId
• sizeBytes
• fileName
• mimeType
• status
• annotationApplied

The screenshot bytes are delivered out-of-band over Ansight’s binary transfer protocol. The resident host uses transferId to map the incoming binary frames to the screenshot artifact.

Example:

{
  "toolId": "ui.get_screenshot",
  "arguments": {
    "format": "jpeg",
    "quality": 85,
    "maxWidth": 1080,
    "annotateNodeIds": false,
    "afterScreenUpdates": true
  }
}

Inspect Node

Arguments:

• nodeId: required node identifier
• includeAncestors: include ancestor chain
• includeDescendants: include descendants
• includeProperties: include implementation-specific properties

Returns:

• platform
• capturedAtUtc
• node
• optional ancestors
• optional descendants

Example:

{
  "toolId": "ui.inspect_node",
  "arguments": {
    "nodeId": "root/stack[0]/button[1]",
    "includeAncestors": true,
    "includeDescendants": false,
    "includeProperties": true
  }
}

Show Overlay

ui.show_overlay creates one or more input-transparent highlight rectangles. It can target a visual-tree nodeId, an array of explicit rects, or one top-level x / y / width / height rectangle.

Arguments:

• overlayId: optional caller-provided id
• nodeId: optional visual tree node id to highlight
• rects: optional array of rectangles with x, y, width, height, and optional label
• x, y, width, height: optional single rectangle coordinates
• label: optional label for a single rectangle
• coordinateSpace: window or visualTree; defaults to window
• strokeColor: overlay stroke color; supports hex colors and common color names
• fillColor: optional fill color; use none or transparent for no fill
• strokeWidth: stroke width
• cornerRadius: rectangle corner radius
• durationMs: overlay lifetime in milliseconds; defaults to 5000, use 0 to persist until removed
• metadata: optional small scalar dictionary for caller context

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.show_overlay",
  "arguments": {
    "nodeId": "root/stack[0]/button[1]",
    "strokeColor": "#FF2D55",
    "fillColor": "transparent",
    "strokeWidth": 3,
    "cornerRadius": 6,
    "durationMs": 5000,
    "metadata": {
      "reason": "tap-target-review"
    }
  }
}

Get Overlay

Arguments:

• overlayId: required overlay id

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.get_overlay",
  "arguments": {
    "overlayId": "tap-target-review"
  }
}

Query Overlays

Arguments:

• metadataKey: optional metadata key that must be present
• metadataValue: optional value that must match metadataKey

Returns:

• platform
• capturedAtUtc
• count
• overlays

Example:

{
  "toolId": "ui.query_overlays",
  "arguments": {
    "metadataKey": "reason",
    "metadataValue": "tap-target-review"
  }
}

Update Overlay

ui.update_overlay edits an existing overlay by id. Omitted fields preserve the current overlay value. Supplying nodeId, rects, or rectangle coordinates replaces the existing geometry.

Arguments:

• overlayId: required overlay id
• nodeId: optional replacement visual tree node id to highlight
• rects: optional replacement rectangles
• x, y, width, height: optional replacement single rectangle coordinates
• label: optional label for a single replacement rectangle
• coordinateSpace: window or visualTree; defaults to window
• strokeColor: optional replacement stroke color
• fillColor: optional replacement fill color; use none or transparent to clear fill
• strokeWidth: optional replacement stroke width
• cornerRadius: optional replacement corner radius
• durationMs: optional replacement lifetime from now; use 0 to persist until removed
• metadata: optional metadata dictionary
• metadataMode: merge, replace, or clear

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.update_overlay",
  "arguments": {
    "overlayId": "tap-target-review",
    "strokeColor": "#34C759",
    "durationMs": 0,
    "metadataMode": "merge",
    "metadata": {
      "status": "confirmed"
    }
  }
}

Remove Overlay

Arguments:

• overlayId: required overlay id

Returns:

• platform
• capturedAtUtc
• overlayId
• removed
• overlay: removed overlay snapshot, or null when no overlay matched

Example:

{
  "toolId": "ui.remove_overlay",
  "arguments": {
    "overlayId": "tap-target-review"
  }
}

Clear Overlays

Arguments:

• metadataKey: optional metadata key that must be present for an overlay to be cleared
• metadataValue: optional value that must match metadataKey

Returns:

• platform
• capturedAtUtc
• count
• overlays: removed overlay snapshots

Example:

{
  "toolId": "ui.clear_overlays",
  "arguments": {
    "metadataKey": "reason",
    "metadataValue": "tap-target-review"
  }
}