Quality

A quality definition answers a per-run question such as “did login stay responsive?” or “did memory flatten after the guide loaded?” It measures telemetry inside a flow and compares the result with an explicit numeric expectation.

Quality evaluation is deterministic. It runs in the host after the functional scenario, not inside the model prompt.

Create a quality check

Quality definitions are JSON files beneath ansight/quality:

{
  "schemaVersion": 1,
  "id": "login-responsiveness",
  "appId": "com.example.app",
  "flow": "login",
  "required": true,
  "missingDataOutcome": "fail",
  "measurements": [
    {
      "id": "fps-p10",
      "channel": {
        "type": "fps"
      },
      "statistic": "p10",
      "expect": {
        "gte": 55
      },
      "minimumSamples": 10,
      "maximumSampleGapMs": 1000
    },
    {
      "id": "low-fps-time-ratio",
      "channel": {
        "type": "fps"
      },
      "statistic": {
        "type": "timeBelowRatio",
        "threshold": 50
      },
      "expect": {
        "lte": 0.05
      },
      "minimumSamples": 10,
      "maximumSampleGapMs": 1000
    }
  ]
}

This example requires the slowest 10% of FPS samples to remain at or above 55 and limits time below 50 FPS to 5% of the flow.

Definition fields

Measurements

A measurement selects a telemetry channel by type, name, source, or kind. Set match to exactlyOne when ambiguity should invalidate the result, or any when matching channels may be combined.

Set platform to android, ios, macos, windows, or other when a measurement only applies to that platform. Non-matching measurements are skipped, which supports platform-native channels such as Android RSS and iOS Physical Footprint without reporting the other platform as missing data.

Supported scalar statistics include:

• min, max, average, p05, p10, p50, and p95;
• memoryPeakIncreaseMiB and memoryRetainedIncreaseMiB;
• tailSlopeMiBPerSecond and tailSpreadMiB; and
• timeBelowRatio with a numeric threshold.

Expectations use gte, lte, or absoluteLte. A memory check can include a pre-flow baseline and a post-flow tail:

{
  "id": "retained-growth-mib",
  "channel": { "type": "memory" },
  "statistic": "memoryRetainedIncreaseMiB",
  "expect": { "lte": 35 },
  "minimumSamples": 5,
  "maximumSampleGapMs": 2000,
  "baselineBeforeStartMs": 5000,
  "tailAfterEndMs": 15000
}

minimumSamples and maximumSampleGapMs prevent sparse evidence from looking more trustworthy than it is. Use baselineBeforeStartMs for growth relative to the pre-flow state and tailAfterEndMs when retained state or post-work stability matters.

Monitor a measurement historically

Add trend to the measurement itself; there is no separate authoring definition:

"trend": {
  "regressionPercent": 20,
  "regressionAbsolute": 15,
  "confirmRuns": 2,
  "baselineRuns": 12,
  "blocking": false
}

The expectation supplies the usual worse direction. If both regression thresholds are present, both must be crossed. Historical monitoring is non-blocking unless blocking is explicitly enabled. See Trends for reference selection, comparable series, and decision states. Co-locating the policy does not combine the runtime decisions: the quality result still describes one run, while the trend result describes its historical comparison.

Run from a workspace test

Add quality IDs to the test definition:

{
  "qualityChecks": ["login-responsiveness"]
}

Checks run after the functional scenario and before an app launched by the runner is stopped. Required failures and inconclusive required evidence affect the test outcome. JSON results contain the full quality report.

Analyze ordinary sessions automatically

Link the App ID to its trusted workspace once:

ansight app register com.example.app --codebase /path/to/workspace

When a matching session finishes, the host evaluates definitions for flows that were actually observed. Unrelated flows are optional skips. Per-session reports are stored with the local session capture as quality-results.json.

Choose useful limits

Begin with representative simulator and physical-device baselines. A threshold tighter than normal device variance creates noise rather than useful evidence. For a new check, consider monitoring with required: false and missingDataOutcome: "warn", then tighten it after reviewing real samples.

ansight test validate .
ansight test quality-history --measurement login-responsiveness.fps-p10

Related docs

• Flows
• Trends
• Tests
• Telemetry Review