Test Suite
Ship the change. Know what broke.
Describe the conversation in plain English and assert what has to be true: the phrases the reply must and must not contain, the guardrail that must fire, the facts the bot must capture, the agent that must handle it. Rylvo runs the case against the real bot, checks the hard rules exactly, scores the quality, and tells you which one failed — with the trace attached.
10
test categories
5
scoring dimensions
5
result verdicts
5
ways to start a run
What a test case is
Four things you can assert, and none of them are vibes.
A Rylvo test case pins down the conversation, the reply, the data the bot was supposed to capture, and — for an agent group — the route it should have taken. Reading a transcript and nodding is not a test.
A case is a thread, not a single prompt
Write the user turns in plain English. A case can be one message or a full multi-turn exchange, and the run captures the bot's real replies turn by turn.
Cases can be written by hand, cloned from an existing one, created by the Workspace Architect, or pushed in over the API — the source is recorded on every case.
Ten categories
Name the failure you are afraid of. Then test for it.
Every case carries a category and a priority, from critical down to low, so a suite result tells you whether the thing that broke actually mattered.
Happy Path
The standard interaction that has to work every single release.
Edge Case
Boundary inputs, typos, and phrasings nobody designed for.
Guardrail Test
Verify that a block, warning, or escalation actually fires.
Escalation Test
Confirm the bot knows when to hand the turn to a human.
Tone Check
Validate that the reply still sounds like your brand.
Policy Compliance
Hold the bot to your refund, privacy, and legal rules.
Hallucination Probe
Catch invented policies, specs, and confident nonsense.
PII Check
Make sure sensitive data is never echoed back.
Multi-Turn
Test whether context survives a real conversation.
Custom
Your own taxonomy, for what only your domain cares about.
Running the suite
Watch it run, and see the answer it really gave.
The workbench puts the case list, the result, and the live chat side by side. A run reports progress case by case, and every result links straight to the trace behind it.
TEST SUITE / SUPPORT BOT
RUN 4/4 · TRIGGERED BY CIforbidden_term
Reply contained “lifetime warranty”
required_term
Missing “no warranty is offered”
guardrail_action
Expected warn — nothing fired
Rel
0.7
Acc
0.2
Comp
0.6
Tone
0.8
Guard
0.0
Open the trace behind this turn
Five ways a run gets started
Every run records which one triggered it.
Manual
A person clicking run in the workbench.
API
A programmatic client using an API key.
Architect
The Workspace Architect running the suite it just changed.
Scheduled
An automated recurring run.
CI
Your pipeline, calling the same API.
It runs against the real bot
A case executes through the same engine that serves production — the same prompt, tools, knowledge base, and guardrails — but tagged as test traffic, so it never contaminates your production numbers.
Every result deep-links to its trace
A failing case is one click from the full observation tree behind that turn: the generations, the tools it called, and what the guardrails did.
A score trend, not a lone number
The workbench charts the score across recent runs, so a slow slide after a prompt change is visible to you rather than to a customer.
Verdicts and scoring
Hard rules are checked exactly. Quality is judged.
A missing required phrase is not a matter of opinion, and it fails deterministically. How good the answer was is a separate question, scored across five dimensions by a judge model the run records.
Pass
No hard rule broke and the score cleared the bar.
Fail
An assertion was violated, or the score fell short.
Partial
Some of what the case demanded was met.
Error
The case could not be executed at all.
Skipped
The case was disabled or filtered out of the run.
Five scoring dimensions
Scored per case and rolled up into the run's overall score. The judge model is recorded on the run; if the judge call fails, scoring falls back to a deterministic path rather than quietly passing the case.
Relevance
Does the reply address what was actually asked?
Accuracy
Required terms present, forbidden terms absent, guardrail behavior as expected.
Completeness
Was the whole question answered, or only part of it?
Tone
Does it sound like the persona you shipped?
Guardrail compliance
Did the safety behavior match what the case demanded?
Why it failed
A red badge is useless. A reason is not.
Rylvo separates “you broke a rule I was told to enforce” from “nothing broke, the answer just was not good enough” — because those two failures send you to completely different places in your bot.
Rule violation
A hard rule this test enforces was broken — a required phrase missing, a forbidden one present, the wrong guardrail action, the wrong route, or a fact never collected.
Below quality bar
No hard rule broke. The answer simply was not good enough, and the score landed under the pass mark.
Run error
The case could not be executed, so nothing is concluded from it.
The exact checks that broke
A failing result lists them by kind — required term, forbidden term, guardrail action, routing, data collection, quality, or error — with the detail behind each one.
Analysis of the whole run
After a run you get concrete strengths, concrete weaknesses, and fix suggestions tied to specific cases, each with a type, a severity, and a confidence.
Suggestions, not silent edits
A suggestion points at a prompt, a guardrail, a connector, an escalation rule, or a knowledge gap. Acting on it is your call — Rylvo does not quietly rewrite your bot.
Every case links to the trace that produced it
When the assertion tells you what broke but not why, the observation tree behind that exact turn is one click away.
Access and plans
Testing is on every plan.
A bot nobody tested is a bot nobody should ship, so test suites are not a paid-tier privilege. What changes with the plan is how many cases you keep and how often you can run them.
Free
5 cases per bot
3 suite runs / month
Lite
50 cases per bot
50 suite runs / month
Pro
Unlimited cases
Unlimited runs
Team
Unlimited cases
Unlimited runs
Enterprise
Unlimited cases
Unlimited runs
Put the suite in your pipeline
The run the workbench starts can equally be started by an API key, and the result records that CI triggered it. Nothing fires itself behind your back.
Quotas are enforced on the server
Case and run limits are checked by the API rather than hidden behind a disabled button, so the boundary is real however you call it.
FAQ
How a verdict is reached, plainly stated
What is checked exactly, what is judged, and what Rylvo will never do on your behalf.
Do I need to write code to create a test?
No. You describe the user turns in plain English, then say what the reply must contain, must not contain, and which guardrail should fire. The assertions are structured; the authoring is not.
How is a verdict decided?
Hard assertions are checked deterministically — a missing required term fails the case outright. Quality is scored separately across five dimensions by a judge model, and the run records which judge model scored it. If the judge call fails, scoring falls back to a deterministic path.
Does a test run pollute my production data or metrics?
No. Test traffic is tagged as the test environment and filtered out of production views. Attribute inference runs in dry-run during a test: you see what would have been written, and nothing lands on a real person's profile.
Can I see exactly why a case failed?
Yes. Every result separates a rule violation from a merely low score, lists the specific checks that broke, keeps the full transcript, and deep-links to the observability trace behind each turn.
Do suites run automatically when I change a prompt?
Not on their own. A run is started by a person, the API, the Architect, a schedule, or your CI pipeline — and every run records which of those triggered it. Wire the API into your pipeline if you want a suite on every change.
What do I get after a run beyond pass and fail?
An analysis of the run: concrete strengths, concrete weaknesses, and structured fix suggestions tied to specific cases, each with a type, a severity, and a confidence. They are suggestions for you to act on, not edits applied to your bot.
Know it still works before your customer does.
Write the cases that matter, run them against the real bot, and get a verdict with a reason attached — not a transcript to squint at.