AI Chatbot Pilot Program: A 4-Week Plan to Scale or Stop
Scope one use case, set targets from your own baseline, soft launch on a few pages, then decide to scale, iterate or stop using a written rule.
- Updated
- Reading time
- 20 min read
11 sections
- Why pilots fail
- The 4-week pilot at a glance
- Week 1: Scope and metrics
TL;DR:
- RAND's 2024 report on AI project failure says that, "by some estimates," more than 80% of AI projects fail, about twice the rate of IT projects without AI. RAND cites that figure from other sources. Its own research is interviews with 65 data scientists and engineers, who named a misunderstood or miscommunicated problem as the most common root cause.
- The four-week plan: Week 1 scope and metrics, Week 2 build and offline test, Week 3 soft launch on a few pages, Week 4 measure and decide.
- Pick the narrowest possible use case. "Return policy questions only" is a real pilot. "AI support" is not.
- Write your targets and a stop rule from your own baseline before launch. If the pilot breaks the stop rule, you stop it instead of redesigning it mid-pilot.
An AI chatbot pilot program needs four weeks and one written decision at the end. Week 1 scopes a single use case and sets targets, Week 2 builds and tests the bot offline, Week 3 puts it in front of a small slice of real visitors, and Week 4 compares the results with the targets and decides whether to scale, iterate or stop.
A "pilot" can be a real test or a way to look busy. The difference is whether you set up the conditions for an honest answer at the end. This plan is opinionated on purpose, because an opinionated plan is one you can finish.
Why pilots fail
Most chatbot pilots fail for organizational reasons, not model reasons. RAND's 2024 report The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (Ryseff, De Bruhl and Newberry, RR-A2680-1, published August 13, 2024, checked November 12, 2026) states that "by some estimates, more than 80 percent of AI projects fail," which it describes as twice the failure rate of IT projects that do not involve AI. That number is an estimate RAND quotes, not a rate RAND measured. RAND's own work is interviews with 65 experienced data scientists and engineers, and it reports five root causes:
- Stakeholders misunderstand or miscommunicate the problem the AI should solve.
- The organization lacks the data needed to train an effective model.
- The team focuses on new technology instead of a real user problem.
- The infrastructure to manage data and deploy models is inadequate.
- The problem is too difficult for AI to solve.
In a support chatbot pilot, those causes usually show up as four practical problems:
Scope creep. The pilot starts as "answer return policy questions" and ends as "be a full support agent across five channels in three languages." The team runs out of time before it finishes anything.
No success criteria. The pilot launches without a specific target. At the end, the team argues about whether a 68% helpful rating is good or bad because nobody wrote down the target.
Wrong stakeholders. Engineering builds the bot. Support never agrees to its handoff rules. Marketing never agrees to its tone. By Week 4 there is no internal champion who will defend the result.
No rollback plan. The bot underperforms, but it is already on the main support page and nobody owns turning it off. So it stays, half-broken, eroding trust in the team that built it.
Every step of this plan exists to neutralize one of those failure modes.
The 4-week pilot at a glance
| Week | Goals | Specific actions | Decision point |
|---|---|---|---|
| 1 | Scope and metrics | Pick one use case, write targets from your baseline, build a 25-question test set, pick the pilot pages | Sign-off from support lead and exec sponsor |
| 2 | Build and offline test | Build the bot, add approved knowledge, run the test set, set handoff rules | Test-set scores at or above the written targets |
| 3 | Soft launch | Put the bot on the pilot pages only, read conversations daily, fix the top 3 failure modes | Quality and handoff numbers moving toward targets |
| 4 | Measure and decide | Compare actuals with targets, check the stop rule, decide scale, iterate or stop | Written decision with the numbers |
Each week has a hard end-of-week artifact. If you cannot produce the artifact, you do not move on. This is how the plan resists scope creep.
Week 1: Scope and metrics
Week 1 decides whether the pilot can produce an honest answer. You choose one use case, write targets from your own numbers, assemble a test set and choose the pilot pages. Most pilots that fail in Week 4 actually failed here.
Define success metrics
Write down three targets and one stop rule before you build anything. Base each one on your own baseline. There is no universal benchmark for a narrow support pilot, and a target copied from a vendor blog says nothing about your queue.
- Quality target. The share of rated answers marked helpful, or the wrong-answer rate in a reviewed sample of conversations. Start from your current help-center feedback, or from how many of 25 test answers your team accepts as correct.
- Handoff target. The share of conversations that go to a person. Start from how many in-scope questions need account access or human judgment today.
- Workload or speed target. In-scope tickets your team still handles, or first reply time. Start from last month's numbers.
- Stop rule. For example: "Stop if any safety failure reaches a customer, or if the wrong-answer rate in the reviewed sample is above our limit at the end of Week 3."
An illustrative example, not a benchmark: a team whose return-policy articles get 72% helpful votes might target at least 70% helpful for the bot, at most 30% of conversations handed off, and at most 5% wrong answers in a 100-conversation review.
Write these in a doc and get sign-off from the support lead and the exec sponsor. This is the document you will read on the Friday of Week 4. For more metric definitions, see chatbot metrics to track.
Match each metric to where you can measure it
Pick metrics your platform can actually report, or plan a manual method before launch. The general definitions work on any platform. The right column lists what Chatsy shows today, so you can see which numbers need your own review or tools.
| Metric | General definition | Where it comes from in Chatsy |
|---|---|---|
| Helpful share | Thumbs-up ratings ÷ all ratings | Helpful rating % on the agent overview (thumbs up or down) |
| Handoff rate | Conversations handed to a person ÷ all conversations | Handoff rate on the agent overview |
| Wrong-answer rate | Wrong answers ÷ conversations reviewed | Not a dashboard metric. Read conversations and count. |
| First reply time | Time from visitor message to first reply | Average first reply time in Analytics |
| Unanswered topics | Questions the bot could not answer well | Knowledge gaps in Analytics: "Rated not helpful", "No answer found", "Handed off" |
| CSAT or containment | Survey score, or conversations closed without a person | Not available in Chatsy. Use your own survey or conversation review. |
Pick the narrowest possible use case
This is where pilots die. The instinct is to build "an AI support agent." The discipline is to build "a return policy bot."
A good Week 1 use case looks like this:
- One question type or one closely related cluster (returns, shipping times, password resets)
- Answered from one or two approved sources (a help-center section, a policy page)
- Resolvable in under four turns
- A clear handoff target when the bot cannot help
Bad Week 1 use cases:
- "Replace tier 1 support"
- "Answer anything on the website"
- "Handle billing, returns, and technical questions"
A narrow scope is not a humble scope. It is the only scope that finishes on time. For a step-by-step setup of a single pilot queue, see how to automate customer support.
Identify the 25-question golden test set
Pull 25 real questions from your support inbox that match the scope. Anonymize them. Include 10 common questions, 5 edge cases, 5 policy-adherence questions, 3 questions where the answer is genuinely not available (the bot should say so and offer a person), and 2 with a hostile or confused tone.
You will use this same test set every week for the rest of the pilot.
Pick the deployment surface
Decide where the bot will live in Week 3 before you build it. Options, from lower risk to higher risk:
- Internal-only test (your own team is the user)
- Help-center pages for the scoped topic only
- Specific product pages (for example, the returns page)
- A share of sessions across the site (needs session sampling in your own code if your platform has no percentage rollout)
- Main support page
Most pilots should pick option 2 or 3. The main support page is for after the pilot.
End-of-week-1 artifact: A one-page brief listing the use case, the three targets with their baselines, the stop rule, the 25-question test set and the pilot pages. Signed by the support lead and the exec sponsor.
Week 2: Build and offline test
Now you build. The constraint is that the bot has to pass the test set before it sees a real customer.
Build the bot
Configure the platform, add the approved knowledge and write the instructions. For a narrow use case this is a few days of work, not two weeks. A walkthrough of the build steps is in build your first AI agent, and the Chatsy specifics are in train your AI agent.
Resist the urge to add features. A pilot bot does not need 15 integrations. It needs to be correct on the use case you scoped.
Run the 25-question test set
Score every question on four dimensions: accuracy, tone, safety and handoff quality. (For the full method, see our chatbot evaluation guide.)
Set the pass criteria from your Week 1 targets, with one fixed rule: safety is 100%, and a bot that fails a safety question does not ship. For the other dimensions, write the threshold as a count. If you require 23 of 25 answers to be accurate, that is 23 ÷ 25 = 0.92, or 92%. Two wrong answers is the most you can accept.
If you do not pass, you iterate. You do not move to Week 3 with a failing offline score and hope production fixes it.
Set up handoff rules
Define when the bot hands off and to whom. These general rules work on most platforms:
- Hand off on any explicit request ("I want to talk to a human")
- Hand off when the bot cannot find a supporting source
- Hand off when the visitor repeats the same question or says an answer did not help
- Hand off on anything outside the scoped use case
In Chatsy, the Live chat module (Hobby plan and above) adds an "Escalate to Human" tool to the agent. Chatsy also offers a handoff or follow-up when the visitor asks for a person, repeats a question or says an answer did not help, when no supporting source is found, when the answer says it is unsure, or when an error occurs. Chatsy does not score sentiment, and it has no staff online or away status, so a handoff always goes to the queue. If nobody watches the queue during pilot hours, turn on email follow-up. Setup details are in the human handoff docs.
Make sure agents know the pilot is happening, what scope the bot covers and how to spot handoffs.
End-of-week-2 artifact: Test-set scores, the instructions and knowledge sources documented, the handoff rules written down, and an agent briefing complete.
Week 3: Soft launch
Week 3 puts the bot in front of real visitors on a small, controlled surface. The goal is to find the failures your test set missed while rollback is still easy.
Limit the launch to a small slice of traffic
Do not put the bot on your whole site on Day 1 of Week 3. Limit it to specific pages, or sample sessions in your own site code. Some platforms offer percentage rollouts or URL-targeting rules. Chatsy does not, so with Chatsy you add the embed script only to the pilot pages (install the widget).
The point of the limited rollout is that you can roll back instantly when something goes wrong. In Chatsy, turning off "Allow public chats" stops the embed script from loading, which works as a same-day kill switch. During Week 2 you can test on the private preview page (/preview/{agent ID}) before public chats are on.
Monitor real conversations daily
Every morning, read 20 to 30 conversations from the previous day. Look for:
- Wrong answers (the bot said something inaccurate)
- Bad handoffs (the bot handed off when it should have answered, or kept going when it should have handed off)
- Tone misses (the bot was breezy with a frustrated customer)
- Unscoped questions (customers asking about things outside the pilot scope)
The first week of real traffic always brings phrasings you did not test, typos, and questions that look like your scope but are actually different. Track every surprise in a doc. In Chatsy, the Conversations page and the Knowledge gaps list in Analytics are the two places to start.
Fix the top 3 failure modes
By Wednesday of Week 3, rank the failure modes by frequency and fix the top three. Do not try to fix all of them. The fixes are usually:
- Adding missing knowledge content
- Adjusting the instructions to handle a specific phrasing
- Tightening or loosening a handoff rule
Re-run the 25-question test set after each fix to make sure you have not regressed.
End-of-week-3 artifact: A list of every conversation reviewed, the top failure modes ranked, the fixes shipped and the latest test-set scores.
Week 4: Measure and decide
This is the week the pilot exists for. Do not let it become "Week 3 part 2."
Compare actuals against criteria
On Monday of Week 4, pull the numbers:
- Helpful share from thumbs ratings
- Handoff rate
- Wrong-answer rate from your conversation review
- First reply time and in-scope ticket volume
- Plan cost and usage for the month
- CSAT, only if you ran your own survey
Put them next to the targets you wrote down in Week 1. Do not negotiate with the targets. The point of writing them down was to remove the temptation to move the goalposts.
Decide: scale, iterate, or kill
Three outcomes are possible.
Scale. You hit all three targets and did not break the stop rule. Write the rollout plan: which pages next, which scopes next, who owns it after the pilot. Get the rollout plan signed off the same week.
Iterate. You hit one or two targets and missed one by a small margin. Iterate is the most dangerous outcome because it has no natural end. Set a hard date (no more than four additional weeks) and a new gate. If you miss again, stop.
Kill. You missed two or three targets by clear margins, or you broke the stop rule. Turn the bot off the same day and write the post-mortem the same week.
The post-mortem is not punishment. It is the most valuable artifact of a failed pilot, because it tells the team what to scope differently next time.
End-of-week-4 artifact: A written decision (scale, iterate or kill) with the supporting numbers, the rollout or shutdown plan, and the next pilot scope if applicable.
Worked example: reading a 400-conversation pilot
Here is how the Week 4 decision works with real arithmetic. All inputs below are assumptions for illustration, not results from a Chatsy customer. Replace them with your own counts.
| Input (assumed) | Value |
|---|---|
| Conversations on the pilot pages in Week 3 and early Week 4 | 400 |
| Conversations with a thumbs rating | 120 |
| Thumbs-up ratings | 90 |
| Conversations handed to a person | 88 |
| Conversations read in the review sample | 100 |
| Wrong answers found in the review | 6 |
| Safety failures | 0 |
The calculations, against the illustrative Week 1 targets:
- Rating rate: 120 ÷ 400 = 0.30, so 30% of conversations were rated.
- Helpful share: 90 ÷ 120 = 0.75, or 75%. Target at least 70%: met.
- Handoff rate: 88 ÷ 400 = 0.22, or 22%. Target at most 30%: met.
- Wrong-answer rate: 6 ÷ 100 = 0.06, or 6%. Target at most 5%: missed by 1 point.
- Safety: 0 failures. Stop rule not broken.
Check how solid each number is before you act on it. With 120 ratings, one rating moves the helpful share by 100 ÷ 120 = 0.83 percentage points, so the 5-point margin over target equals 0.05 x 120 = 6 ratings. With 100 reviewed conversations, the 1-point miss is 0.01 x 100 = 1 conversation.
Decision: iterate. Two targets are met and the miss is one conversation in a small sample. Fix the sources behind the 6 wrong answers, re-run the 25-question test set, review another 100 conversations, and set a date two weeks out. If the wrong-answer rate is still above 5% on that date, stop the pilot.
Pilot vs production cost expectations
Price the pilot from your own volumes and the vendor's billing unit, not from a rule of thumb. Billing units differ: some vendors charge per resolution or per conversation, while Chatsy deducts credits for each AI response, from 1 to 30 credits per response depending on the model (message credits). Models on the Hobby and Standard plans cost at most 10 credits per response.
Using the same 400 conversations, assume 3 AI responses per conversation and 4 runs of the 25-question test set, since test chats are AI responses too:
- Pilot responses: 400 x 3 = 1,200
- Test-set responses: 25 x 4 = 100
- Total: 1,200 + 100 = 1,300 AI responses in the month
| Model cost per response | Credits needed | Hobby ($40, 500 credits, 1 seat) | Standard ($150, 4,000 credits, 3 seats) | Pro ($500, 15,000 credits, 5 seats) |
|---|---|---|---|---|
| 1 credit | 1,300 x 1 = 1,300 | Not enough | Fits | Fits |
| 10 credits | 1,300 x 10 = 13,000 | Not enough | Not enough | Fits |
The credit cap is strict: Chatsy blocks a response that would make usage reach the monthly allowance, so Hobby covers at most 499 one-credit responses. There are no top-ups, overages or rollover, and credits reset monthly. The Free plan ($0, 40 credits) has no live chat, so a pilot with human handoff needs Hobby or above. Seats include the owner, so only Standard and above let teammates work the handoff queue. See pricing for the full plan list.
On Standard with a 1-credit model, the plan price spread over the pilot is $150 ÷ 400 = $0.375 per pilot conversation. That is not your full cost. The 88 handed-off conversations still take agent time. Compare the total with your own fully loaded cost of handling the same 400 conversations by hand, using your payroll and ticket data. Support automation ROI walks through that calculation.
Not for you
Three kinds of pilots should not happen.
The vanity pilot. Leadership wants to say "we have AI." Nobody owns the metrics. The bot launches to a quiet corner of the site and nobody mentions it again. This is not a pilot, it is a press release. Kill it now and save the budget.
The delay tactic. "Let's pilot it for a quarter" usually means "let's not commit yet." If the pilot keeps getting extended without new evidence, the answer is not another extension. The answer is to call the question.
The pilot nobody can staff. If no one can read conversations daily in Week 3 or answer handoffs, the pilot will measure your staffing gap, not the bot. Wait until someone owns those hours.
A real pilot has a scope, a metric, a deadline and a kill switch. If yours does not, fix that before you spend another week on it.
FAQ
Q: Do 80% of AI projects really fail?
The figure is an estimate, not a measurement. RAND's 2024 report says that "by some estimates, more than 80 percent of AI projects fail," twice the rate of IT projects without AI, and cites other sources for that number. RAND's own interviews with 65 data scientists and engineers found that the most common root cause is a misunderstood or miscommunicated problem, followed by missing data. Treat the 80% as a warning about scope and data, not as a forecast for your pilot.
Q: Four weeks seems short. Can we do an 8-week pilot?
Yes, but the structure is the same. Add the weeks to Week 2 (build) or Week 3 (soft launch), not to Week 1 or Week 4. Longer scoping rarely improves outcomes, and longer measurement rarely changes the decision.
Q: What if our success criteria turn out to be wrong?
Acknowledge it in writing during Week 4. For example: "We targeted 75% helpful and reached 68%, and every unhelpful rating was about exchanges, which our help center does not cover." Then run a second pilot against the new criteria. Do not silently move the targets mid-pilot.
Q: Who owns the pilot?
One named person, full stop. Usually a support lead, with engineering and product as supporting players. If three people share ownership, nobody owns it.
Run the four weeks and get an answer
A pilot is not a marketing exercise. It is a structured way to find out whether a chatbot is worth scaling at your company. Done well, it takes four weeks and produces a clear yes or no. Done poorly, it produces a permanent half-deployment that costs the team trust.
If you run the pilot in Chatsy, you can train the agent from web pages, files or written answers, test it privately on the preview page, add the embed script only to your pilot pages, turn on human handoff with the Live chat module on Hobby and above, and track conversations, helpful ratings, handoff rate and knowledge gaps. Chatsy does not offer percentage rollouts, URL-targeting rules, a domain allowlist, CSAT surveys or containment reports, so plan those steps in your own site code or tools. Start free to test answers, or see pricing for the plans that include handoff.