Traces + Analytics
Know exactly what your agent did.
Every turn opens a trace: the nested tree of generations, tool calls, retrievals, guardrails, and agent handoffs behind the answer, with its tokens, cost, and latency. Inspect one turn, build the dashboard you actually need, alert on a threshold, or export it all to your own OpenTelemetry collector.
10
observation types
20
widget metrics
12
dashboard templates
8
score sources
From a turn to a trace
Every turn leaves a record you can actually open.
Rylvo observability is the workspace around one store of traces, observations, and scores. It is built for the question you ask at 2am — what did this bot do, why, and what did it cost — not for a wall of charts nobody drills into.
A turn runs
Any turn on the engine — dashboard test chat, a channel, or the public API — opens a trace tagged with its bot, session, end user, and environment.
The work is recorded as observations
Generations, tools, retrievers, agents, chains, guardrails, evaluators, spans, and events are written as a nested observation tree under that trace.
Scores attach to the result
Numeric, categorical, boolean, or text scores land on the trace from guardrails, LLM judges, evals, code, reviewers, operators, or the end user.
One store serves both surfaces
Traces, observations, and scores share one Postgres-backed store, so the inspector and the analytics widgets never read different numbers.
Teams act on it
Inspect and replay, comment, pin to a dataset, alert on a threshold, or stream the traces to your own OpenTelemetry collector.
Traces workspace
One page, three ways to look at the same traffic.
Read production as individual turns, as conversations, or as the people driving them — then narrow with filters that are honest about whether they searched the loaded window or the whole store.
TRACES / SUPPORT BOT
PRODUCTION · MOST RECENT 500route.classify
160ms
kb.search
190ms
pii.output_filter
12ms
compose.reply
540ms
Scores
Every turn as a row you can scan
The list is the entry point: status, bot, latency, tokens, cost, and environment on one line, with a split-pane detail on the right.
The list loads the most recent 500 traces per source. That window is stated in the UI rather than implied away — use the date range or the server-side filters to reach older traces.
Trace inspector
Open the turn and see every decision inside it.
A trace is not a log line. It is a nested tree of generations, tool calls, retrievals, guardrails, and agent handoffs, with the payloads, timings, tokens, and cost that produced the answer your customer read.
Agent swimlane
When a turn ran as a group of agents, each agent gets its own lane with its handoffs and confidence.
- Per-agent input and output
- Handoff events between agents
- Orchestrator generations called out
- Appears only when the turn is multi-agent
DAG and tree
Read the same observation graph either as a timeline DAG or as a nested tree, whichever matches how you are thinking about the failure.
- Ten observation types
- Nested parent/child spans
- Latency per observation
- Full input/output payload per node
Raw JSON and copy-out
Drop to raw JSON, or copy a plain-text trace report covering performance, guardrails, resources, and evolution rules.
- Preview or JSON per payload
- Copy trace ID in one click
- Copy the report into any model
- Deep-link a trace with ?traceId=
Comments with mentions
Discuss a specific trace in place and @-mention a teammate instead of pasting a screenshot into chat. A commentId deep link scrolls straight to the thread.
Pin to a dataset
Send an interesting trace into a dataset from the detail panel, or multi-select rows and save them in bulk — that is how a production failure becomes a test case.
Scores from eight sources
Customer, user, operator, human reviewer, LLM judge, guardrail, code, and eval scores sit on the trace as numeric, categorical, boolean, or text values.
Failure modes
Derived turn outcomes — no reply, hallucinated action, and similar — are surfaced as a filterable outcome rather than something you have to eyeball.
Analytics you build
Not six fixed charts. The question you actually have.
Analytics is a widget builder over the same trace store. Pick a view, a metric, a breakdown, and a chart type; filter it; drop it into a room. Start from a template when you would rather not start from nothing.
Compose a widget from four choices
Add filters across eight columns, cap a breakdown with top-N, overlay a trend line, or compare bots side by side.
4
data views
Traces, tool calls, cost, runs
20
metrics
Count, cost, tokens, latency percentiles, pass/fail/warn/block/error rates, unique users
12
breakdowns
Bot, model, environment, status, user, channel, source, day/hour/week, bot comparison
7
chart types
Line, vertical bar, horizontal bar, big number, pie, histogram, pivot
Rooms hold the dashboards your team keeps coming back to
Save a widget to the org library, reuse it across rooms, and scope a room to a bot, an environment, and a date range.
Alerts, cost, and export
Watch the numbers so you do not have to.
Set a threshold on the metrics that actually page someone, price your own models so the cost column matches your bill, and stream everything into the observability stack you already run.
Five alert metrics, per bot or org-wide
A threshold over a window on your own traffic.
Error rate
Share of turns that errored or were blocked
Warning rate
Share of turns with a tool, connector, KB, media, or guardrail warning
p95 latency
95th-percentile turn latency
Cost / turn
Average cost per turn
Spend in window
Total spend over the alert window
Cost you can trust
Every trace carries tokens and cost. Save your own $/M price overrides and the list, detail, and user rollups all re-price the moment you hit save.
OpenTelemetry export
Point an export target at your collector with an endpoint, headers, and a batch size. Traces leave in GenAI semantic-convention format (gen_ai.*), and the target reports its last run and count.
Ingest health
An ingest badge tells you when trace writes are failing, so a quiet dashboard reads as a broken pipe rather than a quiet week.
Retention and access
Keep the evidence exactly as long as your policy says.
Trace retention is a plan window you can shorten, viewing and deleting are separate permissions, and a trace nearing expiry says so before it disappears.
Included trace retention by plan
An organization can set a shorter window than its plan allows; it cannot silently extend past it.
Free
7 days
Trace list only — full replay needs a paid plan
Lite
14 days
Full trace replay
Pro
30 days
Full trace replay
Team
90 days
Full trace replay
Enterprise
Unlimited
Full trace replay
Viewing is not deleting
Trace deletion is its own capability. Reviewers and on-call engineers can read production traces without being able to erase them.
Expiry is visible
Traces approaching the end of the retention window are filterable, so anything worth keeping can be pinned to a dataset before the purge runs.
Environments stay separate
Test and production traffic are tagged at the source and split everywhere they are displayed, so a dashboard is never quietly padded with test runs.
Free sees the list; full replay starts on a paid plan
Free organizations keep 7 days of traces and can watch what is happening. Opening a turn to its full observation tree, scores, and payloads is a paid capability.
FAQ
What is captured, what is bounded, plainly stated
The boundaries worth knowing before you rely on Rylvo traces as your record of what production did.
Is every run captured, or only a sample?
Every turn on the engine opens a trace with its observation tree. What is bounded is retention and what the screen loads: traces expire on your plan's retention window, and the Traces list loads the most recent 500 per source, which the UI states directly.
Can I keep test traffic out of my production numbers?
Yes. Each trace carries its environment, and the Traces page and analytics widgets both filter to test, production, or all. A session containing both is flagged instead of silently averaged.
What is the difference between a trace and an analytics widget?
A trace is one turn: its observation tree, scores, tokens, cost, and latency. A widget aggregates many traces along a metric and a breakdown. They read the same store, so a widget and the trace behind it cannot disagree.
Can I get traces into my own tooling?
Yes. Configure an export target with an endpoint, headers, and a batch size, and traces are sent in OpenTelemetry GenAI semantic-convention format (gen_ai.*) to your collector. The export shows when it last ran and how many traces it sent.
Can Rylvo tell me when something breaks without me watching?
You can define threshold alerts per bot or org-wide on error rate, warning rate, p95 latency, cost per turn, or spend in a window. Alerting is a threshold on your own traffic — not an anomaly-detection promise.
Who can delete a trace?
Deletion is a separate capability from viewing. A member can be allowed to inspect traces without being allowed to remove them, and retention purges expired traces on its own schedule.
Ship an agent you can see into.
Traces start the first time a bot answers. Open a turn, read its observation tree, score it, alert on it, and send it to your own collector.