Rylvo Datasets: Curate, Preserve, and Analyze AI Conversation Traces with Independent Retention Policies

Every conversation your AI bot has is a potential learning opportunity. A failed guardrail is a signal that your safety rules need refinement. A high-latency turn reveals a performance bottleneck. An escalation to a human operator shows where the bot's knowledge gaps are. A perfectly handled complex query is a goldmine for fine-tuning. But these signals are buried in a firehose of daily chat traffic. Without a systematic way to collect, preserve, and organize the traces that matter, they are lost to your organization's retention window.

Most observability platforms let you search traces, maybe star a few favorites, and export them to a CSV. That is fine for ad-hoc debugging but completely inadequate for building structured datasets for evaluation, fine-tuning, compliance, or product analysis. You need named collections with independent retention policies. You need to pin traces from anywhere in the platform into curated sets. You need to inspect full trace details inline without navigating away. You need quota enforcement so your preservation budget is respected. And you need the entire system to sync with your observability backend so preserved traces are never accidentally pruned.

Rylvo Datasets provides all of this. It is a trace curation system that lets you create named datasets, pin conversation traces into them from any page in the platform, assign independent retention policies per dataset, and inspect full trace details inline — all with transactional quota enforcement and automatic export-lock syncing.

In this guide, we will explore the complete Datasets system: what datasets are, the three retention policies, how pinning works, the dashboard layout, quota enforcement, use cases for evaluation and compliance, and how to create your first dataset in minutes.

What Are Datasets? Named Collections of Pinned Traces

A dataset is a named, curated collection of conversation traces. Think of it as a folder where you save the most important, interesting, or problematic conversations for future reference, analysis, or export.

Unlike ad-hoc trace searches, datasets are persistent. They survive browser sessions. They have their own retention policies. They can be shared across your team. And they integrate with every part of the Rylvo platform.

You create a dataset with a name and description, optionally choose a retention policy, and then pin traces into it from the Traces dashboard, Mission Control, or anywhere else traces appear. Each pinned trace stores a lightweight snapshot — bot name, verification outcome, latency, user message preview, and source — so you can quickly scan the dataset contents without loading full traces.

When you open a trace from a dataset, the full trace detail renders inline using the same viewer as the Traces dashboard. For canonical observability traces, you get the full replay with timeline, observations, and scores. For dashboard chat traces, you get the message history with metadata. The viewer supports both trace types with automatic source resolution and fallback.

Three Retention Policies: Control What Stays and For How Long

The core power of Datasets is the ability to assign independent retention policies to each collection. This means different datasets can have different lifetimes based on their purpose.

Keep Forever

The strongest preservation option. Every trace pinned to this dataset is permanently protected from the organization's retention pruning. This is ideal for compliance archives, legal hold collections, critical incident records, and gold-standard evaluation sets that you never want to lose. The export lock is set to a far-future date, and the observability backend is instructed to never delete these traces.

Keep Extra Days

A middle-ground option that extends trace retention beyond the organization's default window. You specify a number of extra days — for example, 180 days beyond the org's 30-day default. This gives you a 210-day total retention for traces in this dataset. This is ideal for seasonal analysis, quarterly review collections, and extended debugging sessions where you need traces longer than the default but not permanently.

Inherit Org Retention

The grouping-only option. Traces in this dataset are not given any special retention treatment. They live and die according to the organization's global retention policy. This is ideal for temporary working sets, exploratory collections, and draft evaluation sets where you just want organization and filtering without consuming your preservation quota.

When you change a dataset's retention policy, the system automatically recomputes the organization's preserved trace count. If switching from non-pinning to pinning would exceed your quota, the transaction is blocked with a clear error message and upgrade prompt.

How Pinning Works: From Firehose to Curated Collection

Pinning a trace to a dataset is a one-click action available anywhere traces appear in the Rylvo platform.

From the Traces dashboard, select one or more traces and click Save to Dataset. The system performs a pre-flight quota check so you know before confirming whether you have headroom in your preserved trace budget. From Mission Control, pin an escalated or anomalous conversation directly into a dataset for later review. From the bot chat page, pin a particularly good or bad conversation for analysis.

When a trace is pinned, the system creates a lightweight item document containing: the trace ID, source type (canonical observability or dashboard chat), who pinned it and when, and a snapshot with bot name, verification outcome, latency, and user message preview. This snapshot makes dataset browsing fast without loading full traces.

If the dataset has a pinning retention policy, the system checks the organization's preserved trace quota transactionally. If the trace is not already pinned by another pinning dataset, the preserved count is incremented. If it would exceed the quota, the pin is blocked with a typed QuotaExceededError that the UI surfaces as a helpful message.

The Dashboard: Sub-Sidebar, Summary Cards, and Inline Inspection

The Datasets dashboard at /dashboard/datasets provides a complete curation workspace.

Collapsible Sub-Sidebar

The left side shows a collapsible sidebar listing all your datasets. Each row displays the dataset name, trace count, last updated timestamp, and retention policy badge. Keep Forever datasets get a blue badge. Inherit datasets get a neutral badge. A search box filters the list by name or description. A New Dataset button opens the creation panel. Each row has a kebab menu on hover for Edit, Archive, and Delete actions.

Main Content Area

The main area shows a usage strip at the top: preserved traces used versus maximum, and datasets used versus maximum. This gives immediate visibility into your quota consumption.

When you select a dataset, a summary card appears with compact stat chips: trace count, retention policy, and last updated. Below the card is a two-pane layout. The left pane lists all pinned traces with bot name, verification outcome badge, latency, timestamp, and user message preview. Click any trace to open it in the right pane. The right pane renders the full trace detail inline, with a back button on mobile. If no trace is selected, the right pane shows a helpful prompt to select one.

Create and Edit Panel

The side panel slides in from the right for creating or editing datasets. It captures the dataset name, optional description, and retention policy with three radio options. The Keep Extra Days option reveals a number input for the extra day count. Saving validates the name length, checks quotas transactionally, and syncs export locks with the observability backend.

Quota Enforcement: Hard Limits with Clear Feedback

Datasets enforces two quotas to ensure organizations stay within their plan limits.

Max Datasets controls how many active datasets an organization can have. When you create a new dataset, the system checks this quota transactionally. If you are at your limit, the creation is blocked with a clear upgrade prompt.

Max Preserved Traces controls how many unique traces can be pinned by pinning datasets. This is a deduplicated count — a trace pinned to three Keep Forever datasets only counts once against the quota. The system checks this quota on every pin operation, both for single traces and bulk adds. When performing a bulk add from multi-select, the system stops at the first quota block and reports how many traces were added versus blocked.

Quota pre-flight checks let the UI surface helpful warnings before the user clicks confirm. For example, if you select 20 traces to save but only have headroom for 15, the modal shows exactly how many will succeed and how many will be blocked.

When quotas are exceeded, the system fires analytics events so administrators can track usage trends and plan upgrades.

Export Lock Syncing: Observability Integration

Datasets does not just store pointers in Firestore. It actively syncs with the observability backend to ensure preserved traces are never pruned.

When a trace is pinned to a pinning dataset, the system computes the maximum export lock date based on all datasets containing that trace. Keep Forever traces get a far-future lock date. Keep Extra Days traces get a lock date computed from the organization's default retention plus the extra days. This lock is sent to the observability backend via a PUT to the trace lock endpoint.

When a trace is unpinned, the system recomputes the lock date based on remaining pinning datasets and updates the backend. When a dataset's retention policy changes, all contained traces have their locks recomputed and synced. This ensures the observability pruning system always knows which traces must be preserved, regardless of how datasets evolve over time.

Use Cases: What Teams Build with Datasets

Evaluation Sets for Agent Evolution

ML engineers create datasets of high-quality and low-quality traces, label them by outcome, and use them to evaluate rule effectiveness, guardrail accuracy, and bot response quality. These datasets are kept forever or for extended periods to support longitudinal evaluation.

Compliance and Legal Hold

Legal and compliance teams create datasets of conversations involving sensitive topics, escalations, or regulatory interactions. These are set to Keep Forever with strict access controls, providing an immutable archive for audit and legal discovery.

Failed Guardrail Analysis

Operators create datasets of conversations where guardrails fired, blocked, or failed. By grouping these together, they can identify patterns in guardrail evasion, false positives, and edge cases, feeding directly into guardrail refinement.

High-Value Conversation Curation

Sales and support teams create datasets of their best-handled conversations. These become training examples for new bot configurations, onboarding materials for human operators, and showcases for stakeholder demos.

Performance Regression Tracking

Engineers create datasets of slow or expensive traces. By grouping high-latency turns, they can correlate performance issues with model changes, connector additions, or knowledge base updates, enabling targeted optimization.

Comparison: Datasets vs. Trace Search and Export

Getting Started

Step 1: Open the Datasets Dashboard

Navigate to /dashboard/datasets. The sub-sidebar lists any existing datasets. The usage strip shows your current quota consumption.

Step 2: Create Your First Dataset

Click New Dataset. Enter a name like "Escalated Conversations" or "Guardrail Failures." Add an optional description. Choose a retention policy. For a working set, use Inherit. For a compliance archive, use Keep Forever. Save.

Step 3: Pin Traces from the Traces Dashboard

Go to /dashboard/traces. Select one or more traces. Click Save to Dataset and choose your new dataset. The system checks quota and pins the traces with snapshots.

Step 4: Inspect Traces Inline

Return to /dashboard/datasets. Select your dataset. Click any trace in the left pane to see the full detail in the right pane. Review the conversation, verification outcome, latency, and metadata.

Step 5: Monitor Quota Usage

Watch the preserved traces counter at the top of the page. If you are approaching your limit, archive old datasets or upgrade your plan.

FAQ

What is a Rylvo Dataset? A named, curated collection of conversation traces with an independent retention policy. Datasets let you pin important traces for evaluation, compliance, analysis, and fine-tuning.

What retention policies are available? Three: Keep Forever (permanent preservation), Keep Extra Days (extend beyond org default), and Inherit Org Retention (grouping only, no special protection).

How do I add traces to a dataset? Select traces from the Traces dashboard, Mission Control, or bot chat page and click Save to Dataset. You can add one trace or many at once.

What happens when I archive a dataset? The dataset is hidden from the list but the traces remain pinned. Archiving decrements the dataset count but does not affect preserved traces.

What happens when I delete a dataset? The dataset and all its item pointers are permanently removed. Traces that were only pinned by that dataset lose their preservation protection and may be pruned by the org retention policy.

How does quota enforcement work? Two quotas: max datasets and max preserved traces. Creating a dataset or pinning a trace checks the relevant quota transactionally. Bulk adds stop at the first quota block.

What is a preserved trace? A unique trace that is a member of at least one dataset with a pinning retention policy. The same trace pinned to three pinning datasets still counts as one preserved trace.

Does pinning affect the observability backend? Yes. Pinning a trace to a pinning dataset sets an export lock on the observability backend, preventing automatic pruning. Unpinning removes the lock if no other pinning dataset contains the trace.

Can I change a dataset's retention policy? Yes. Edit the dataset and select a different policy. The system recomputes preserved trace counts and syncs export locks automatically.

Who can create datasets? Operators, administrators, and owners can create and manage datasets. Viewers can browse and inspect.

Ready to Start Curating Your Conversations?

Rylvo Datasets turns the flood of daily bot conversations into structured, searchable, and preservable collections. Create a dataset for every purpose: evaluation, compliance, debugging, training, and analysis. Pin traces with one click from anywhere in the platform. Set retention policies that match your needs. Inspect full trace details inline. Stay within your quotas with transparent enforcement.

Your most valuable conversations do not have to disappear with your retention window. Pin them. Preserve them. Learn from them.

Open Datasets and create your first collection today.

Pin what matters. Preserve what counts. Analyze what teaches.