Edge Cases
Catch what your agent gets wrong.
Surface the failures hiding in real traces, cluster them into one triageable problem, and put a rule in front of them — a guardrail that blocks, warns, rewrites, escalates, or simply watches. Then attack your own bot with thirteen red-team strategies, and keep the regression test that proves the fix held.
12
failure categories
10
guardrail types
13
red-team strategies
12
scheduled scan types
The loop
Find the failure, guard against it, prove it stays fixed.
Every agent fails in ways nobody designed for. Edge Cases is the closed loop around that: surface the failure, cluster it, encode a rule that stops it, apply the fix, and keep the regression test that stops it coming back quietly.
Find it
An edge case arrives from a scan of real traces, an adversarial red-team run, a guardrail that fired, or an operator filing it by hand.
Cluster it
The same failure hitting fifty times is one edge case with an occurrence count and its traces attached — not fifty alerts to triage.
Guard against it
A guardrail encodes the rule: block, warn, rewrite, escalate, or simply log. It is evaluated on every turn, against the input or the output.
Fix it
A fix proposal spells out the change. Guardrail and prompt fixes can be applied from the proposal; the rest arrive as explicit steps.
Prove it stays fixed
Regression test cases are linked to the fix, so you can re-run the exact failure the proposal claims to have solved.
What counts as an edge case
Twelve ways an agent lets you down.
Each one carries a severity from critical down to info, a status as it moves from open to resolved, an assignee, and the traces that prove it happened.
EDGE CASES / SUPPORT BOT
3 OPENRedact email addresses before the reply is sent
Proposed guardrail · pii_detection · block
Hallucination
The answer was not grounded in the knowledge the bot had.
Policy violation
A company or workflow rule was broken.
Missed escalation
It should have gone to a human, and it did not.
Wrong tool
The bot picked the wrong tool for the job.
Missing tool call
It should have called a tool, and answered from memory instead.
Wrong stage
The workflow stage was misidentified.
Stale memory
An out-of-date session fact was used in the reply.
Unknown intent
The user wanted something no workflow covers.
Tone violation
The reply did not sound like your brand.
PII leak
Sensitive personal information was exposed.
Loop detected
The conversation went round in circles.
Custom
The failure only your domain would recognize.
An edge case is a cluster, not an alert
Recurring failures are grouped into a single record with an occurrence count, a first-seen and last-seen time, and every trace that produced it — so triage is a queue of problems rather than a firehose of incidents.
Guardrails
A rule that runs on every single turn.
A guardrail is not a prompt asking the model nicely. It is a condition evaluated against the real turn, with a consequence you chose, counted every time it fires.
Ten kinds of guardrail
A guardrail is a named rule attached to one bot or to the whole workspace, with a priority that decides who wins when two of them disagree.
Input filter
Screen what the user sent before the bot acts on it
Output filter
Check the reply before the customer ever sees it
Fact check
Hold the answer against the knowledge base
Policy check
Enforce a company rule
PII detection
Catch personal data on its way out
Tone check
Keep the voice on brand
Loop breaker
Break a repeating conversation
Escalation override
Force a handoff when conditions are met
Tool gate
Decide which tools may be called, and when
Custom
Your own rule
When a guardrail runs is decided by the field its condition reads — an input condition is checked before the turn, an output condition after it. The type is how you label the rule, not what schedules it.
Red team
Attack your own bot before somebody else does.
A red-team run drives adversarial conversations against your real bot and files what it breaks as edge cases. It executes as test traffic, so it never pollutes your production numbers.
Thirteen built-in strategies, plus your own
Each strategy carries an expected severity, so a policy probe that gets through outranks a bot that merely got chatty.
Policy probe
Tries to talk its way past your business rules
Hallucination bait
Asks about companies and events that do not exist
Tool confusion
Nudges the bot toward the wrong tool
Escalation boundary
Pushes on the edge of when a human is needed
Injection attempt
Classic prompt injection
Context overflow
Stresses memory and the context window
MCP abuse
Leans on the connected MCP tools
PII extraction
Tries to get personal data back out
Multi-turn coercion
Wears the bot down across a conversation
Jailbreak
Roleplay and persona attacks
Function leak
Fishes for your tool definitions
Billing bypass
Goes after paid capability for free
KB poisoning
Tries to plant false knowledge
From finding to fix
A proposal you review — not a change you discover later.
Rylvo drafts the fix and, for the two kinds it can make safely, applies it once you approve. It will not quietly rewrite your bot behind your back.
Guardrail add
Applied for you
The proposed guardrail is created and enabled from the proposal.
Prompt edit
Applied for you
The prompt change is written, with the before and after visible first.
Threshold adjust
Steps for you
The proposal states what to tune, and tells the workspace it landed.
Escalation rule
Steps for you
The new escalation condition arrives as an explicit instruction.
Workflow expand
Steps for you
The gap in the workflow is described, along with what should cover it.
Custom
Steps for you
Whatever the fix is, written as steps somebody can follow.
The fix comes with a regression test
A proposal links test cases covering the failure it claims to solve, and you can run them from the proposal — so “fixed” is something you verified, not something you were told.
An edge case has a life, not just a badge
Open, acknowledged, fix proposed, fix applied, resolved, or dismissed — with an assignee, and a record of who closed it.
Running it on a schedule
Nobody watches a dashboard at three in the morning.
Put the scan on a timer instead. Twelve task types run on a cadence in your timezone and bring you what they found, through the channel you actually read.
Twelve scheduled scans
Hourly through monthly, or a precise interval, weekday, or day of the month at a wall-clock time you set.
Trace scan
Sweep recent traces for edge cases
Red team run
Attack the bot on a schedule
Guardrail audit
Review whether your guardrails are still earning their place
Conversation quality
A model reviews tone, helpfulness, and unresolved issues
KB gap analysis
Find questions answered badly and propose knowledge to add
Cost & usage watch
Flag spend anomalies and projected overruns
Connector / MCP health
Audit tool failures, latency, and auth problems
Complaint watch
Watch complaint-risk conversations and escalations
Dev review
Surface failed runs and unresolved patterns
Test suite run
Nightly regression against your test cases
Report generation
Build and distribute the recurring report
Custom
Whatever your team needs on a timer
Delivered where you read it
Findings go to Slack, email, a webhook, Discord, or Microsoft Teams — not to a page you have to remember to open.
Some scans think, most just count
Conversation quality, cost and usage, KB gaps, and connector health are reviewed by a model. The rest are deterministic checks, and Rylvo does not dress them up as intelligence.
What it finds becomes an edge case
A scheduled scan does not just email you a number. What it surfaces enters the same triage queue, with the same clustering, guardrails, and fixes behind it.
FAQ
What fires, what blocks, and what Rylvo changes on its own
The boundaries worth knowing before you let a guardrail stand between your bot and your customer.
Where do edge cases come from?
Four places: a scan of real production traces, an adversarial red-team run, a guardrail that fired, or an operator filing one by hand. The source is recorded on every edge case.
Does the same failure create a hundred alerts?
No. Recurring failures are grouped into one edge case with an occurrence count, the traces that produced it, and first-seen and last-seen timestamps.
When does a guardrail actually run?
The condition decides. A condition on the input is evaluated before the bot acts; a condition on the output is evaluated before the reply is delivered. Conditions can also read tool calls, tool results, the stage, and metadata.
Will a fix be applied to my bot without me?
No. A proposal is reviewed by a person. Guardrail and prompt fixes can then be applied from the proposal itself; other fix types arrive as written steps for you to carry out.
How do I know a fix actually worked?
A fix proposal links regression test cases covering the failure it addresses, and you can run them from the proposal — so a closed edge case is one you verified, not one you were told about.
Is red-teaming safe to run against a live bot?
Red-team runs execute as test traffic, so the adversarial conversations are tagged as the test environment and stay out of your production metrics.
The failure you have not found is the one shipping right now.
Scan the traces, attack your own bot, put a rule in front of the failure, and keep the regression test that proves it is gone.