Observability

In one line. Every run leaves a trace — what ran, what it read, what it called, what it cost, what it decided — and knowing where each trace lives turns “the AI did something weird” into a five-minute diagnosis. You’ll be able to. Answer the three operational questions on any live solution: is it working now, what exactly happened on that run, and is quality holding over time.

Is it working right now?

Start where the work is:

• The Crew tab on a collection (A12) — the live view: status pill, pipeline rail, each crew member lighting up as it works, recent decisions and runs. If a demo or go-live has a single pane of glass, this is it.
• The workspace (E2) — lifecycle stages are a queue-depth readout: a growing In review pile or documents stuck in New tells you where the process is jammed.
• Dashboards (E5) — throughput, exceptions and aging over time, for the people who own the process rather than the plumbing.

What exactly happened on that run?

Every execution is recorded, at the level you need:

Trace	Where	What it answers
Agent run trace	The agent’s Runs (and the playground, live)	The full loop: inputs, tool calls, output, per-field confidence, token cost, latency
XFlow runs	Studio ▸ Automation ▸ XFlow ▸ (flow) ▸ Runs	Which node ran, with what parameters, what each produced, where it failed
Mesh baton trail	The crew view for mesh-worked collections	Which member handed what to whom, and where the baton stopped
Document history	The document’s detail pane (E3)	What was extracted, corrected, decided — and by whom or what
Chat & search audit	Studio ▸ Search ▸ Audit and Studio ▸ Chat ▸ Audit	What users asked, what was answered, what was retrieved

The habit that pays: when an output looks wrong, open the run trace first. Nine times out of ten the trace shows the cause directly — a missing attachment, a tool that returned nothing, a low-confidence field that should have routed to review.

Is quality holding over time?

Point-in-time traces don’t catch slow drift. Two instruments do:

• Evaluations (A12) — scheduled scored runs against golden datasets, compared to a pinned baseline. This is your regression gate for prompt, model and pipeline changes.
• Review-rate trends — the share of documents routing to human review is a quality signal in itself: rising review rates mean confidence is dropping somewhere upstream, and the review queue tells you which fields.

Cost is part of observability

Run traces carry token usage and cost. When a solution gets expensive, the traces show which agent, which step, and usually why — an over-attached knowledge tab or a heavier model than the job needs (A1 has the diagnosis table). Measure before switching models; the playground makes the comparison cheap. Environment-wide spend lives in the admin console below.

The platform-level console

Everything above is per-project. Platform admins (the System Admin role) get a whole-environment console at Admin ▸ General ▸ Observability — and /admin itself lands on the Action Center, a self-clearing to-do list of setup gaps, incidents, and cost breaches. The pages that matter most:

Page	What it answers
Pulse	Live platform health at a glance.
Pages · APIs · DB Ops · Jobs · Exceptions	Performance and error drill-downs — from a slow route down to the offending query or stack trace.
Cost	LLM gateway spend from the durable daily rollup — usage, health, and spend per workspace and model.
Infra	What the environment actually runs on: Azure inventory with live health, node/DB/cache metrics, and the real cloud bill (month-to-date + forecast), pulled read-only from Azure. Local stacks show an honest estimate instead.
Budgets · Retention	Spend envelopes that warn before the invoice, and the data-retention dials.

The escalation path, in order

1. Crew tab — is work moving?
2. The stuck document’s history — where did it stop?
3. The run trace — what did the worker actually do?
4. The audit surfaces — what have users been experiencing?
5. Evaluations — did a recent change move quality?

Learn this ladder once and most “something’s wrong” reports resolve without an engineer.

Where to go next

• The live view in depth: A12 · Crew & evaluations.
• The queues humans work: G2 · Human-in-the-loop.
• The record that satisfies auditors: G5 · Audit & compliance.