Understand what your app is showing

VisualTree tools connect what you see on screen with the elements that make up the live UI. Use hierarchy snapshots, focused node inspection, and screenshots to investigate layout or content problems.

Install

dotnet add package Ansight.Tools.VisualTree

Register the Suite

using Ansight;
using Ansight.Tools.VisualTree;

var options = Options.CreateBuilder()
    .WithVisualTreeTools()
    .WithReadOnlyToolAccess()
    .Build();

Registration API

• WithVisualTreeTools(): registers the full visual tree, screenshot, and diagnostic overlay suite. No suite-specific configuration is currently required.
• WithReadOnlyToolAccess(): exposes visual tree, screenshot, node inspection, and read-only overlay tools.
• WithReadWriteToolAccess(): also exposes overlay mutation tools for drawing, updating, and clearing diagnostic overlays.

Visual-Tree Providers

Visual-tree capture is provider-based. VisualTreeProviderRegistry always exposes the platform hierarchy as native, and Ansight.Tools.Maui registers a separate maui source.

Additional frameworks can implement IVisualTreeProvider and register a named source:

using Ansight.Tools.VisualTree;

using var registration = VisualTreeProviderRegistry.Register(
    new CustomVisualTreeProvider());

The provider supplies a stable Source, a display name, GetVisualTreeAsync(...), and InspectNodeAsync(...). Pass source to ui.get_visual_tree or ui.inspect_node; omitting it selects native. VisualTreeProviderRegistry.Query() returns every registered source. Provider responses use the shared compact visual-tree v2 contract.

Local features can reuse the registry without enabling remote tools. For example, Annotated Feedback captures every registered source.

Specific Concerns

• The tools inspect the foreground scene only.
• Screenshot capture exports rendered UI content, so treat it as sensitive.
• ui.get_screenshot requires a live binary transfer channel from the paired host. The result returns transfer metadata, not inline base64.
• includeComputedStyles and detailed node inspection can reveal UI labels and accessibility metadata.
• Overlay tools draw input-transparent diagnostic rectangles over the active native window.
• Overlay mutation tools require write access. Persistent overlays created with durationMs = 0 stay visible until removed or cleared.

Important: Calling screenshot tools will result in an FPS drop while the current frame is captured, encoded, and transferred. Avoid screenshot-heavy investigations during trend measurements unless visual evidence is required.

Tool Matrix

Get Visual Tree

Arguments:

• includeBounds: include node bounds, default true
• includeComputedStyles: include implementation-specific properties, default false
• maxDepth: child depth limit, default 8
• rootNodeId: optional subtree root
• source: optional provider source; defaults to native
• maxNodes: maximum number of nodes to capture

Returns:

• platform
• source
• capturedAtUtc
• format
• types: capture-local type-name registry referenced by node typeId
• root

Each compact-v2 node includes semantic role, supported actions, interactability, stable automation id when available, platform-neutral presentation data, and structural child order. See the Visual-Tree Payload Contract before consuming the JSON directly.

Example:

{
  "toolId": "ui.get_visual_tree",
  "arguments": {
    "includeBounds": true,
    "includeComputedStyles": false,
    "maxDepth": 6
  }
}

Get Screenshot

For the full comparison between this on-demand tool and periodic session JPEG capture, see JPEG Screenshots.

Arguments:

• format: png or jpeg
• quality: JPEG quality 1-100
• maxWidth: optional width cap
• annotateNodeIds: overlay node ids on the screenshot
• afterScreenUpdates: Apple platforms wait for pending screen updates before rendering; default true

Returns:

• platform
• capturedAtUtc
• format
• width
• height
• deliveryMode = websocket_binary
• wireProtocol = ansight.file-transfer.v1
• transferId
• downloadId
• sizeBytes
• fileName
• mimeType
• status
• annotationApplied

The screenshot bytes are delivered out-of-band over Ansight’s binary transfer protocol. The resident host uses transferId to map the incoming binary frames to the screenshot artifact.

Example:

{
  "toolId": "ui.get_screenshot",
  "arguments": {
    "format": "jpeg",
    "quality": 85,
    "maxWidth": 1080,
    "annotateNodeIds": false,
    "afterScreenUpdates": true
  }
}

Inspect Node

Arguments:

• nodeId: required node identifier
• includeAncestors: include ancestor chain
• includeDescendants: include descendants
• includeProperties: include implementation-specific properties

Returns:

• platform
• capturedAtUtc
• node
• optional ancestors
• optional descendants

Example:

{
  "toolId": "ui.inspect_node",
  "arguments": {
    "nodeId": "root/stack[0]/button[1]",
    "includeAncestors": true,
    "includeDescendants": false,
    "includeProperties": true
  }
}

Show Overlay

ui.show_overlay creates one or more input-transparent highlight rectangles. It can target a visual-tree nodeId, an array of explicit rects, or one top-level x / y / width / height rectangle.

Arguments:

• overlayId: optional caller-provided id
• nodeId: optional visual tree node id to highlight
• rects: optional array of rectangles with x, y, width, height, and optional label
• x, y, width, height: optional single rectangle coordinates
• label: optional label for a single rectangle
• coordinateSpace: window or visualTree; defaults to window
• strokeColor: overlay stroke color; supports hex colors and common color names
• fillColor: optional fill color; use none or transparent for no fill
• strokeWidth: stroke width
• cornerRadius: rectangle corner radius
• durationMs: overlay lifetime in milliseconds; defaults to 5000, use 0 to persist until removed
• metadata: optional small scalar dictionary for caller context

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.show_overlay",
  "arguments": {
    "nodeId": "root/stack[0]/button[1]",
    "strokeColor": "#FF2D55",
    "fillColor": "transparent",
    "strokeWidth": 3,
    "cornerRadius": 6,
    "durationMs": 5000,
    "metadata": {
      "reason": "tap-target-review"
    }
  }
}

Get Overlay

Arguments:

• overlayId: required overlay id

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.get_overlay",
  "arguments": {
    "overlayId": "tap-target-review"
  }
}

Query Overlays

Arguments:

• metadataKey: optional metadata key that must be present
• metadataValue: optional value that must match metadataKey

Returns:

• platform
• capturedAtUtc
• count
• overlays

Example:

{
  "toolId": "ui.query_overlays",
  "arguments": {
    "metadataKey": "reason",
    "metadataValue": "tap-target-review"
  }
}

Update Overlay

ui.update_overlay edits an existing overlay by id. Omitted fields preserve the current overlay value. Supplying nodeId, rects, or rectangle coordinates replaces the existing geometry.

Arguments:

• overlayId: required overlay id
• nodeId: optional replacement visual tree node id to highlight
• rects: optional replacement rectangles
• x, y, width, height: optional replacement single rectangle coordinates
• label: optional label for a single replacement rectangle
• coordinateSpace: window or visualTree; defaults to window
• strokeColor: optional replacement stroke color
• fillColor: optional replacement fill color; use none or transparent to clear fill
• strokeWidth: optional replacement stroke width
• cornerRadius: optional replacement corner radius
• durationMs: optional replacement lifetime from now; use 0 to persist until removed
• metadata: optional metadata dictionary
• metadataMode: merge, replace, or clear

Returns:

• platform
• capturedAtUtc
• overlay

Example:

{
  "toolId": "ui.update_overlay",
  "arguments": {
    "overlayId": "tap-target-review",
    "strokeColor": "#34C759",
    "durationMs": 0,
    "metadataMode": "merge",
    "metadata": {
      "status": "confirmed"
    }
  }
}

Remove Overlay

Arguments:

• overlayId: required overlay id

Returns:

• platform
• capturedAtUtc
• overlayId
• removed
• overlay: removed overlay snapshot, or null when no overlay matched

Example:

{
  "toolId": "ui.remove_overlay",
  "arguments": {
    "overlayId": "tap-target-review"
  }
}

Clear Overlays

Arguments:

• metadataKey: optional metadata key that must be present for an overlay to be cleared
• metadataValue: optional value that must match metadataKey

Returns:

• platform
• capturedAtUtc
• count
• overlays: removed overlay snapshots

Example:

{
  "toolId": "ui.clear_overlays",
  "arguments": {
    "metadataKey": "reason",
    "metadataValue": "tap-target-review"
  }
}