A chatbot staging environment is a second copy of your assistant where you change and try things while customers keep talking to the version that works. You edit in Test, freeze the changes you trust into a release, make that release live on purpose, and point back at the previous one if it goes wrong.
This post is part of our guide to testing an AI chatbot before launch.
Key takeaways
- An edit to a prompt or a knowledge source changes every answer that depends on it. Without a test copy, each saved edit is an untested change in front of customers.
- A release should freeze everything that decides how the assistant answers (prompts, knowledge, flows, guardrails) and leave out live data like conversations and secrets.
- Rollback should be a pointer change to a release that was live before, not a rebuild from memory.
- Shared plumbing (webhooks, channels, settings) usually applies at once, so test calls need their own destination.
- Run the same checks before every release: the changed questions, the old questions, and one real channel.
What a chatbot staging environment does
| Question | Without a test copy | With Test and Production |
|---|---|---|
| When does an edit reach customers? | When you click save | When an admin activates a release |
| What does a visitor see mid-edit? | Whatever is saved at that moment | The last release, unchanged |
| Can you ship one change and hold another? | No, everything saved is live | Yes, you pick what goes into each release |
| How do you undo a bad change? | Edit it back by hand and hope you remember the old text | Roll back to the previous release |
| Where do test chats go? | Into the same reports and webhooks as real ones | Into separate traces, analytics and test destinations |
The problem: every save is a deploy
Some chatbot builders still save straight to the live bot: you edit the instructions, press save, and the next visitor gets the new behavior. Others add a publish step, which stops half-written edits going out but still leaves you one working copy rather than a staging copy. Either way, it works until you are halfway through rewriting a refund policy and a customer asks about refunds.
AI assistants make this worse than ordinary websites in three ways:
- Small edits have wide effects. One sentence added to a system prompt ("keep answers short") changes the tone of every reply, including the ones you did not think about.
- Knowledge changes are invisible. Upload a new price list and the old one is still indexed, so the assistant can now quote either. Nothing on screen tells you which.
- You cannot read the output in advance. A web page looks the same to everyone. A model's reply depends on the question, so you only find a regression by asking the questions.
A worked example. A dental practice changes its prompt on Monday to "always offer the next available appointment." The assistant now offers slots even to patients asking about insurance. If saves go live, that shows up in the next real conversation. With Test and Production, you ask the insurance questions in Test first, see the bad replies in the trace, and fix the prompt before it goes into a release. (This is an illustrative scenario, not a measured result.)
Software teams learned this long ago. Google's Site Reliability Engineering book (written in early 2016) reports that "roughly 70% of outages are due to changes in a live system". That is a finding about live systems in general, not about chatbots. Its change-management advice has three parts: progressive rollouts, quick and accurate detection of problems, and safe rollback. A Test/Production split gives you the first and third by design, and the checklist below covers the second (source checked as of November 2026).
When a builder has a publish step but no separate staging copy, the usual workaround is a second bot. Botpress, for example, has a Publish button and keeps a version for each publish, and its documentation also describes copying a bot's configuration into another bot so you can "maintain separate bots for development, staging, and production" (as of November 2026). That works, but you maintain two assistants by hand, and each promotion is a manual copy where a prompt, a knowledge source or a guardrail can be left behind.
How a staging environment works in Rylvo: Test, release, Production
The model is simple once you see it as three boxes:
TEST (you edit here) RELEASE (frozen) PRODUCTION (customers)
--------------------- ------------------- ----------------------
bots, prompts, flows tick exact versions of the activate replies read the
teams, skills, -------> items you ticked --------> active release
guardrails, knowledge, (a "candidate") (read-only)
automations checks: tests, approval
^ |
| ROLLBACK: point Production at any |
+---- refresh Test <-- release that was live before <----------+
SHARED by both, never released: connectors, MCP servers, channels, Settings
NEVER in a release: conversations, traces, secret values
In Rylvo, every workspace on every plan has Test and Production. The switch sits at the top of the main menu, and a Releases card at its foot shows what Test has that customers do not. A release takes two steps:
- Create candidate. A release starts empty and you tick what ships. Create candidate then freezes the selection. From that moment the candidate never changes, and Production has not changed yet. One candidate waits at a time.
- Activate Production. An admin or owner confirms, with a summary of what changes and what stays. Production switches by updating one pointer, and running replies pick it up within about 30 seconds.
Production is read-only for everyone, admins included. Rylvo's AI builder, the Architect, makes changes only in Test and refuses to edit Production, so a request like "make the bot friendlier" can never land on live replies without a release.
What visitors see during an edit
Nothing new. While you rewrite a prompt in Test, the website widget, WhatsApp and every other channel keep answering from the active release. A visitor halfway through a conversation gets the same assistant at the end of it as at the start, unless a release is activated in between.
What moves in a release
A release should hold everything that decides how the assistant answers, and nothing that is live data. In Rylvo that splits like this:
| In a release | Shared by Test and Production (applies at once) | Never in a release |
|---|---|---|
| Bots, with their collection fields, state settings and code steps | Connectors (webhooks and tools) | Conversations |
| The exact prompt versions each bot uses | MCP servers | Traces |
| Guided flows and multi-agent teams | Channels (widget, WhatsApp and the rest) | Secret values |
| Skills and guardrails | Workspace Settings | |
| Knowledge bases, their sources and search settings | ||
| Scheduled tasks and reports |
Three details matter in practice.
Unticked means kept. If you leave a change out, Production keeps exactly what it has: a changed item stays at its live version, a new item is not added, and a deletion is not applied. Deletions are listed in red before you freeze, so you never remove something by accident.
Knowledge is copied, not re-indexed. When a release with knowledge goes live, the passages you tested are copied to Production as they are. What customers search is what you searched in Test. The knowledge base guide covers how to check retrieval before you ship it.
Prompts are versioned. Every prompt version is kept, and Production pages show the live one read-only. If you want to compare two versions on real traffic before you choose, the post on versioning and A/B testing a system prompt walks through it.
The honest caveats
Two things reach customers without a release, and you should know them before you rely on the split:
- Shared plumbing. A change to a connector, an MCP server, a channel or a Setting applies to live replies at once. Treat those edits like production changes: make them at a quiet hour and check a live conversation afterward.
- Agent Evolution rules. Agent Evolution (Pro and Enterprise; the 7-day Pro trial includes it) turns repeated failures into rules. A rule belongs to the bot, not to an environment, so once it is on it applies to Test and Production replies right away. That is why a drafted rule waits for your approval by default and can be replayed on past conversations first. One switch, off by default, lets rules above a confidence you set turn on by themselves. If you switch it on, those rules reach customers with no person in between. Fixes it makes to settings in Test still need a release.
Keep test data apart
A test copy is only half the job if your test chats still land in the real CRM. Three places need separating.
Webhooks and tools. Connectors are shared, so a test conversation could call the same endpoint a real one does. In Rylvo a Test reply calls a connector only after an owner or admin has verified that connector for Test and the workspace allows Test deliveries. Until then the call is blocked and shown as Not sent. Verified Test calls carry a header your endpoint can branch on:
POST /leads HTTP/1.1
Host: hooks.example.com
Content-Type: application/json
X-Rylvo-Environment: test
Route requests with that header to a sandbox table, a test Slack channel or a Zapier test step, and you can try a lead-capture flow end to end without a fake lead reaching sales.
Your own conversation database. Saving conversations to your own database is part of Lite, Pro and Enterprise; the 7-day Pro trial includes it. If you do, and you use Postgres, Supabase, MySQL or MongoDB, test traffic goes to tables with a _test suffix by default (conversations_test next to conversations). On MySQL, PlanetScale and MongoDB you can instead point test traffic at a separate database. Create the test tables before your first test chat on a SQL database such as Postgres or MySQL, or the write fails with an error that says so (MongoDB creates its test collections on first write). The conversation storage guide has the table layouts.
Keys, traces and analytics. An API key belongs to the environment it was made in, and no header switches it, so a script holding a Test key cannot reach Production. Traces and analytics are split by environment too, which keeps test chats out of the numbers you report.
Gate the release on tests
A staging copy helps only if you look at it before you ship. The simplest version is to replay a fixed set of questions against Test before every release. Rylvo Tests does that with scripted conversations: rules are checked by code and quality is scored by a judge model you pick.
An admin or owner can switch on Hold releases under Tests, Release gate, on any plan. Then every bot in a candidate must pass a test run against that exact candidate before it can be activated. The drawer names each bot that does not pass and the tests to fix. Two-person approval, where a second admin approves and the person who created the candidate cannot, is available through the releases API. Custom pass criteria for the gate, and asking the gate from your CI, are part of Enterprise.
For what goes into that test set, use the checklist in the pillar post linked at the top.
How to roll back chatbot changes
A rollback is only safe if it lands exactly where you were. Rebuilding the old prompt from memory, or restoring a knowledge file you think was the right one, is another untested change.
Because a release freezes a complete set of what Production runs, rolling back is a pointer move:
- Open the Releases drawer and go to History. Each release shows who created it and who activated it.
- On any release that was live before, choose Roll back to this release.
- Production points at it again. Nothing is rebuilt or redeployed, and running replies use it within about 30 seconds.
- The release you left is kept as rolled back, so you can return to it the same way once it is fixed.
Admins and owners can roll back; everyone else can read the history. Remember the caveat above: a rollback restores what is in releases. If the problem came from a connector, a channel setting or an Agent Evolution rule, change that directly.
After a rollback, bring Test back in line. Refresh Test pulls the live release into Test and keeps Test edits that differ until you choose to overwrite them. Reset Test, after you type RESET TEST, starts it over. Both save a checkpoint first.
A 7-step release checklist
Run this before every release.
- Read the diff. Open the candidate and list every ticked item. Untick anything you did not mean to ship, and read every deletion in red.
- Ask the changed questions. For each change, ask Test the three or four questions it was meant to improve, phrased the way customers phrase them.
- Ask the old questions. Replay your standing test set so a fix for refunds does not break opening hours. If Hold releases is on, this run is what lets the candidate through.
- Check the knowledge. For any new or replaced document, ask one question only the new version answers, and one the old version answered differently. The reply must use the new figure.
- Trigger every tool once. Run each connector the change touches in Test and confirm the call arrived at your test destination with the test header, not at the real one.
- Activate at a quiet hour, and write one line on what changed and why, so the next person reading History knows.
- Watch the first live conversations. Open traces for the next ten Production replies on the channel that matters most. If one goes wrong, roll back first and investigate in Test.
When a simpler setup is enough
If your assistant answers ten questions from one page, you edit it once a month, and nothing calls an outside system, a second bot or a careful manual check may be all you need. The split earns its keep once more than one person edits the assistant, once it writes to a CRM or booking system, or once a wrong answer costs you a customer.
If you are setting up your first assistant now, start with adding an AI chatbot to your website, and check pricing for what each plan includes. Test and Production come with every plan.
Frequently asked questions
What is a chatbot staging environment?
It is a separate copy of your assistant where you edit and try changes while customers keep talking to the live copy. In Rylvo every workspace has two: Test, where you build, and Production, which customers use and which changes only when an admin or owner activates a release.
Do edits go live immediately?
Not in Rylvo. Changes to bots, prompts, flows, teams, skills, guardrails, knowledge and automations are made in Test and reach customers only in a release. Four things are shared and apply at once: connectors, MCP servers, channels and workspace Settings. An Agent Evolution rule that is turned on also applies to Test and Production replies right away.
How do I roll back a chatbot change?
Open the Releases drawer, go to History and choose Roll back to this release on any release that was live before. Running replies use it within about 30 seconds. It does not undo connector, channel or Agent Evolution changes, because those are not in releases.
Can I test webhooks without sending real data?
Yes. A Test reply calls one of your connectors only after an owner or admin verifies that connector for Test and the workspace allows Test deliveries; otherwise the call is blocked and shown as Not sent. Verified Test calls carry an X-Rylvo-Environment: test header so your endpoint can route them to a test destination.
Do test conversations end up in my analytics or my database?
Not in your live data. Traces and analytics are split by environment, so test chats stay out of the numbers you report. If you save conversations to your own database, test traffic goes to tables with a _test suffix by default, or to a separate test database on MySQL, PlanetScale and MongoDB.
