Rylvo Analytics: Custom AI Dashboards with Real-Time Bot Health, Cost Tracking, and Operational Insights

Building and deploying AI bots is only half the battle. The other half is understanding whether they are working well, costing too much, failing too often, or frustrating users. Without analytics, you are flying blind — guessing at bot quality, estimating costs, and discovering problems only after customers complain.

Most AI platforms offer basic usage counts and maybe a token tally. But bot operations demand much more granular insight. You need to know which bot is failing most often. Which model is consuming your budget. Whether latency is creeping up after a prompt change. Which guardrails are triggering and why. How verification outcomes are distributed across different conversation types. And you need all of this organized in views that make sense for different stakeholders — product teams, engineers, operators, finance, and leadership.

Rylvo Analytics is a comprehensive analytics platform built specifically for AI bot operations. It combines custom analytics rooms with 17 configurable chart types, dual-source data merging from Firestore and backend APIs, real-time cost tracking, product event analytics, and Mission Control operational KPIs — all in one unified dashboard at /dashboard/analytics.

In this guide, we will explore the complete Analytics system: Analytics Rooms with their 17 chart types, the dual-source data architecture, cost analytics, product analytics, operational analytics, room templates for different stakeholders, and how everything ties together to give you complete visibility into your AI operations.

Why Bot Analytics Is Different from Generic Web Analytics

Traditional web analytics tracks page views, click-through rates, and conversion funnels. These metrics tell you how users interact with a website. They do not tell you whether an AI bot hallucinated, whether a guardrail blocked a legitimate request, or whether a model switch increased latency by 300 milliseconds.

Bot analytics requires fundamentally different metrics:

• Verification outcomes — Did the bot's response pass quality checks, trigger warnings, or fail entirely?
• Guardrail triggers — Which safety rules fired, and were they justified?
• Latency distributions — How long does each conversation turn take, and where are the bottlenecks?
• Cost attribution — Which bot, model, or conversation consumed the most tokens and budget?
• Error patterns — What types of failures are recurring, and which bots are most affected?
• Escalation rates — How often do conversations require human intervention?
• Tool execution rates — How often does the bot successfully call external tools versus failing or being blocked?

Rylvo Analytics tracks all of these and more, organizing them into customizable rooms that different teams can tailor to their specific needs.

Analytics Rooms: Your Custom Command Center

At the heart of Rylvo Analytics are Analytics Rooms — named, persisted dashboard views that you can create, customize, and share within your organization. Each room is a self-contained analytics workspace with its own filters and chart configuration.

Room Filters: Scope Your Data

Every room has three filter dimensions that determine what data it displays:

Bot Scope — View all bots or isolate a single bot. This is essential when you manage multiple bots and need to track each one's performance independently.

Environment — Switch between all environments, test only (dashboard playground traffic), or production only (live API traffic). This lets you separate testing noise from real user data.

Date Range — Choose 7 days, 30 days, 90 days, or all time. Different questions require different windows. A developer debugging a recent deployment needs 7 days. A finance team reviewing quarterly spend needs 90 days.

Room Customization: Make It Yours

Each room's charts are fully customizable. You can:

• Add or remove chart cards from 17 available types
• Reorder charts via drag-and-drop in edit mode
• Customize each chart's title, width (half or full), height (compact, normal, or tall), and color accent (8 color options)
• Save changes to persist the configuration in Firestore

Rooms are private to your organization but stored centrally, so any team member with appropriate permissions can access them.

17 Chart Types Across Five Groups

Rylvo Analytics offers 17 distinct chart types organized into five logical groups. Each chart answers a different question about your bot operations.

Core Group: The Numbers That Matter Most

Primary Stats — A compact stat row showing total runs, verification pass count, warning count, failure count, blocked conversations, and average latency. This is the daily health check every bot owner needs.

Secondary Stats — Escalations, executed actions, safety override rate, and other secondary operational indicators. Complements the primary stats with additional context.

Cost Group: Know What AI Costs You

Cost Stats — Total spend, average cost per call, total tokens consumed, input/output token split, and infrastructure cost. The first thing finance looks at.

Cost Over Time — A time-series chart showing daily spend across the selected date range. Spot cost spikes immediately — a sudden jump might indicate a model change, increased traffic, or a runaway tool loop.

Cost by Model — Spend grouped by model ID. Compare the cost efficiency of GPT-4.1 Mini versus Claude 3.5 Sonnet at a glance. Essential for teams evaluating model swaps.

Cost by Bot — Spend grouped by bot. If one bot consumes 80% of your budget while serving 20% of traffic, this chart reveals it instantly.

Token Usage — Prompt versus completion tokens over time. Identify if your bots are becoming more verbose (increasing completion tokens) or if user inputs are growing (increasing prompt tokens).

Trend Group: Watch Patterns Emerge

Runs Over Time — Daily run volume. Track traffic patterns, identify peak usage periods, and correlate spikes with marketing campaigns or product launches.

Error Rate Over Time — Daily error percentage. A rising error rate is often the first sign of a model degradation, prompt issue, or tool failure. Catch it before it affects many users.

Latency Over Time — Daily p50 and p95 latency. p50 tells you the typical experience. p95 tells you the worst-case experience. Together they reveal whether your bots are getting slower and for whom.

Verification Trend — Per-run verification outcomes over time. See the distribution of pass, warn, and fail statuses as a trend rather than an aggregate. Identify when a bot started failing more often.

Breakdown Group: Understand Composition

Verification Distribution — A pie chart of pass, warn, and fail share. One glance tells you the overall quality mix of your bot's responses.

Top Stages — The most frequently hit stage classifications. Understand how conversations flow through your bot's decision pipeline.

Top Actions — The most-selected actions or tool calls. See which tools your bot relies on most and which might be underutilized.

Model Usage — Distribution of models used across runs. Track model adoption and migration progress.

Diagnostics Group: Debug with Data

Latency Histogram — Distribution of run latencies. Identify whether most runs are fast with a few outliers, or whether latency is broadly elevated. Essential for debugging performance issues.

Verifier Reasons — Top reasons for verifier failures or warnings. When verification fails, this tells you why — which guardrails triggered, which conditions matched.

Escalation Reasons — Top reasons conversations were escalated to human operators. Understand the failure modes that require human intervention.

Dual-Source Data Architecture: Firestore + Backend API

Rylvo Analytics merges data from two sources to provide the most complete picture possible:

Firestore Chat Traces — Dashboard playground conversations, test traffic, and widget interactions are stored in Firestore as ChatTraceDoc records. These contain full conversation transcripts, token usage, cost data, guardrail results, MCP tool calls, and latency measurements. Firestore data is available immediately and provides the primary data for cost analytics and dashboard traffic.

Backend API (Postgres + Canonical Observability) — Production API traffic is stored in the backend's PostgreSQL database and the canonical observability system. The backend provides aggregated KPIs, trace run summaries, and historical analytics for production traffic.

The Analytics page merges both sources intelligently:

• Firestore traces are primary for runs and overview metrics
• Backend API data supplements with additional production traffic
• Runs are deduplicated by trace ID to prevent double-counting
• Transactions from billing are merged with synthetic trace transactions for complete cost visibility
• If the backend API is temporarily unavailable, Firestore data renders independently without blocking

This dual-source architecture ensures resilience. A backend outage does not leave you without analytics, and production API traffic is not lost when dashboard traffic dominates.

Product Analytics: 130+ Events Across 20 Categories

Beyond bot operational metrics, Rylvo tracks product analytics through a comprehensive event system. Over 130 typed events across 20 categories capture every significant user action in the platform.

Events are dual-written: one stream goes to Firebase Analytics for standard product analytics, and another stream is batched and written to a Firestore mirror for real-time dashboard queries. This dual-write approach gives you both long-term aggregate analytics and immediate operational visibility.

Categories include marketing (CTA clicks, waitlist signups), dashboard navigation, bot management (creation, testing, deployment), prompt operations, knowledge base interactions, connector usage, billing events, agent evolution activities, edge case handling, mission control operations, and team management.

The Usage Tab in the dashboard visualizes this data: active users and sessions as a 7-day area chart, error rates as a stacked area chart (React, API, Network layers), top events frequency as a vertical bar chart, and a live activity feed showing real-time event streams.

Cost Analytics: Real-Time LLM Spend Tracking

Every bot conversation has a cost. Rylvo Analytics tracks it in real time using the model pricing catalog.

For each trace, the system computes:

• Input cost — Prompt tokens multiplied by the model's input price
• Output cost — Completion tokens multiplied by the model's output price
• Infrastructure cost — Platform overhead for the conversation turn
• Total cost — Sum of all cost components in USD

Cost analytics answer critical business questions: Which bot is most expensive to operate? Did switching from GPT-4 to GPT-4 Mini cut costs as expected? Is a particular conversation type or user behavior pattern driving up spend? Are costs trending up or down over time?

The Cost Analytics Widget on the dashboard home page provides an at-a-glance summary of top bots by cost, top models by cost, total spend, and average cost per request. In Analytics Rooms, the five cost chart types provide deeper drill-downs.

Mission Control Analytics: Operator Performance

For teams running human-in-the-loop operations, Mission Control Analytics tracks operator KPIs:

• Active conversations — How many live conversations need attention right now
• Alerts fired — How many oversight alerts triggered
• MTTA — Mean Time To Acknowledge an alert
• Approvals resolved — How many pending approvals were handled
• MTTR — Mean Time To Resolve an intervention

Charts include interventions per day, alerts per day, intervention mix pie, approval outcomes pie, bot hotspots (top 10 bots by alert count), and an operator leaderboard showing who handled the most interventions with the fastest response times.

These metrics are essential for compliance, staffing decisions, and operational excellence. They answer questions like: Are we responding to alerts fast enough? Which bots generate the most operator workload? Which team members are most effective?

Room Templates for Different Stakeholders

Different teams need different views. Rylvo Analytics supports this through room templates and custom room creation.

Executive Summary Room — All bots, production environment, 7-day window. Primary Stats, Cost Stats, and Verification Distribution. A 30-second health check for leadership.

Cost Tracker Room — All bots, all environments, 30-day window. Cost Stats, Cost Over Time, Cost by Bot, Cost by Model. The finance team's go-to view.

Debug Room — All bots, test environment, 7-day window. Error Rate Over Time, Verifier Reasons, Latency Histogram. Engineers use this to diagnose issues after deployments.

SOC Watch Room — All bots, production environment, 24-hour window. Verification Distribution, Top Stages, Verification Trend. Operations teams monitor this for real-time anomaly detection.

Bot Comparison Room — All bots, production environment, 7-day window. Primary Stats, Model Usage, Latency Histogram. Product teams use this to compare bot versions or configurations.

Comparison: Rylvo Analytics vs. Basic Usage Dashboards

Getting Started

Step 1: Open Analytics

Navigate to /dashboard/analytics. A default "General" room is automatically created on first visit with six core charts: Primary Stats, Cost Stats, Cost Over Time, Runs Over Time, Verifier Pie, and Top Actions.

Step 2: Explore the Default Room

Review the default room's charts. Switch the environment filter between All, Test, and Production to see how data changes. Adjust the date range to see longer or shorter trends.

Step 3: Create Your First Custom Room

Click "New Room" in the sidebar. Name it (e.g., "Production Health"), select production environment, choose a 7-day date range, and select the charts you want. Create the room.

Step 4: Customize Charts

Click "Customize" to open the room configurator. Rename charts, adjust widths and heights, pick color accents, and reorder cards via drag-and-drop. Save your changes.

Step 5: Create Role-Specific Rooms

Create a "Cost Tracker" room for finance with cost-focused charts. Create a "Debug" room for engineering with diagnostic charts. Create a "SOC Watch" room for operations with real-time breakdown charts.

Step 6: Monitor and Iterate

Check your rooms daily. Watch for cost spikes, latency increases, or verification degradation. Use different date ranges to spot trends that short windows hide.

FAQ

What is Rylvo Analytics? A comprehensive analytics platform for AI bot operations featuring custom Analytics Rooms with 17 chart types, cost tracking, product event analytics, and operational KPIs.

What are Analytics Rooms? Named, persisted dashboard views with configurable filters (bot, environment, date range) and customizable chart grids that you can drag, resize, and color-code.

How many chart types are available? Seventeen across five groups: Core (2), Cost (5), Trends (4), Breakdowns (4), and Diagnostics (3).

Where does the data come from? Two sources: Firestore chat traces (dashboard/test traffic) and the backend API (production API traffic). The system merges them intelligently with deduplication.

Can I track costs per bot? Yes. The Cost by Bot chart shows spend grouped by bot. Cost Stats shows total spend, average cost per call, and token breakdown.

Can I separate test and production data? Yes. Every room has an environment filter: All, Test, or Production.

What is product analytics? 130+ typed events tracking user actions across the platform, visualized in the Usage Tab with active users, error rates, top events, and a live activity feed.

What are Mission Control Analytics? Operator-focused KPIs including MTTA, MTTR, intervention trends, alert rates, bot hotspots, and an operator leaderboard.

Can I share rooms with my team? Rooms are stored at the organization level, so any team member with appropriate permissions can access them.

Is the data real-time? The live activity feed updates in real time via Firestore onSnapshot. Room charts are refreshed on load or when filters change.

Can I export charts? Not yet. Scheduled exports and embeddable charts are on the roadmap.

Ready to See Your Bots in Full Detail?

Rylvo Analytics transforms raw bot traffic into actionable intelligence. Custom rooms let every stakeholder see exactly what they need. Seventeen chart types cover every dimension of bot health, cost, and performance. Dual-source data merging ensures completeness and resilience. Product analytics track adoption and usage. Mission Control analytics measure operational excellence.

Whether you are an executive checking weekly health, an engineer debugging a latency spike, a finance manager tracking LLM spend, or an operator monitoring alert response times, Rylvo Analytics has a view for you.

Open Analytics and create your first custom room today.

See every metric. Understand every trend. Optimize every bot.