Skip to content

AI Chatbot Pilot Program: A 4-Week Plan to Scale or Stop

Scope one use case, set targets from your own baseline, soft launch on a few pages, then decide to scale, iterate or stop using a written rule.

Asad AliFounder & CEO
Updated
Reading time
20 min read

TL;DR:

  • RAND's 2024 report on AI project failure says that, "by some estimates," more than 80% of AI projects fail, about twice the rate of IT projects without AI. RAND cites that figure from other sources. Its own research is interviews with 65 data scientists and engineers, who named a misunderstood or miscommunicated problem as the most common root cause.
  • The four-week plan: Week 1 scope and metrics, Week 2 build and offline test, Week 3 soft launch on a few pages, Week 4 measure and decide.
  • Pick the narrowest possible use case. "Return policy questions only" is a real pilot. "AI support" is not.
  • Write your targets and a stop rule from your own baseline before launch. If the pilot breaks the stop rule, you stop it instead of redesigning it mid-pilot.

An AI chatbot pilot program needs four weeks and one written decision at the end. Week 1 scopes a single use case and sets targets, Week 2 builds and tests the bot offline, Week 3 puts it in front of a small slice of real visitors, and Week 4 compares the results with the targets and decides whether to scale, iterate or stop.

A "pilot" can be a real test or a way to look busy. The difference is whether you set up the conditions for an honest answer at the end. This plan is opinionated on purpose, because an opinionated plan is one you can finish.

Why pilots fail

Most chatbot pilots fail for organizational reasons, not model reasons. RAND's 2024 report The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (Ryseff, De Bruhl and Newberry, RR-A2680-1, published August 13, 2024, checked November 12, 2026) states that "by some estimates, more than 80 percent of AI projects fail," which it describes as twice the failure rate of IT projects that do not involve AI. That number is an estimate RAND quotes, not a rate RAND measured. RAND's own work is interviews with 65 experienced data scientists and engineers, and it reports five root causes:

  1. Stakeholders misunderstand or miscommunicate the problem the AI should solve.
  2. The organization lacks the data needed to train an effective model.
  3. The team focuses on new technology instead of a real user problem.
  4. The infrastructure to manage data and deploy models is inadequate.
  5. The problem is too difficult for AI to solve.

In a support chatbot pilot, those causes usually show up as four practical problems:

Scope creep. The pilot starts as "answer return policy questions" and ends as "be a full support agent across five channels in three languages." The team runs out of time before it finishes anything.

No success criteria. The pilot launches without a specific target. At the end, the team argues about whether a 68% helpful rating is good or bad because nobody wrote down the target.

Wrong stakeholders. Engineering builds the bot. Support never agrees to its handoff rules. Marketing never agrees to its tone. By Week 4 there is no internal champion who will defend the result.

No rollback plan. The bot underperforms, but it is already on the main support page and nobody owns turning it off. So it stays, half-broken, eroding trust in the team that built it.

Every step of this plan exists to neutralize one of those failure modes.

The 4-week pilot at a glance

WeekGoalsSpecific actionsDecision point
1Scope and metricsPick one use case, write targets from your baseline, build a 25-question test set, pick the pilot pagesSign-off from support lead and exec sponsor
2Build and offline testBuild the bot, add approved knowledge, run the test set, set handoff rulesTest-set scores at or above the written targets
3Soft launchPut the bot on the pilot pages only, read conversations daily, fix the top 3 failure modesQuality and handoff numbers moving toward targets
4Measure and decideCompare actuals with targets, check the stop rule, decide scale, iterate or stopWritten decision with the numbers

Each week has a hard end-of-week artifact. If you cannot produce the artifact, you do not move on. This is how the plan resists scope creep.

Week 1: Scope and metrics

Week 1 decides whether the pilot can produce an honest answer. You choose one use case, write targets from your own numbers, assemble a test set and choose the pilot pages. Most pilots that fail in Week 4 actually failed here.

Define success metrics

Write down three targets and one stop rule before you build anything. Base each one on your own baseline. There is no universal benchmark for a narrow support pilot, and a target copied from a vendor blog says nothing about your queue.

  • Quality target. The share of rated answers marked helpful, or the wrong-answer rate in a reviewed sample of conversations. Start from your current help-center feedback, or from how many of 25 test answers your team accepts as correct.
  • Handoff target. The share of conversations that go to a person. Start from how many in-scope questions need account access or human judgment today.
  • Workload or speed target. In-scope tickets your team still handles, or first reply time. Start from last month's numbers.
  • Stop rule. For example: "Stop if any safety failure reaches a customer, or if the wrong-answer rate in the reviewed sample is above our limit at the end of Week 3."

An illustrative example, not a benchmark: a team whose return-policy articles get 72% helpful votes might target at least 70% helpful for the bot, at most 30% of conversations handed off, and at most 5% wrong answers in a 100-conversation review.

Write these in a doc and get sign-off from the support lead and the exec sponsor. This is the document you will read on the Friday of Week 4. For more metric definitions, see chatbot metrics to track.

Match each metric to where you can measure it

Pick metrics your platform can actually report, or plan a manual method before launch. The general definitions work on any platform. The right column lists what Chatsy shows today, so you can see which numbers need your own review or tools.

MetricGeneral definitionWhere it comes from in Chatsy
Helpful shareThumbs-up ratings ÷ all ratingsHelpful rating % on the agent overview (thumbs up or down)
Handoff rateConversations handed to a person ÷ all conversationsHandoff rate on the agent overview
Wrong-answer rateWrong answers ÷ conversations reviewedNot a dashboard metric. Read conversations and count.
First reply timeTime from visitor message to first replyAverage first reply time in Analytics
Unanswered topicsQuestions the bot could not answer wellKnowledge gaps in Analytics: "Rated not helpful", "No answer found", "Handed off"
CSAT or containmentSurvey score, or conversations closed without a personNot available in Chatsy. Use your own survey or conversation review.

Pick the narrowest possible use case

This is where pilots die. The instinct is to build "an AI support agent." The discipline is to build "a return policy bot."

A good Week 1 use case looks like this:

  • One question type or one closely related cluster (returns, shipping times, password resets)
  • Answered from one or two approved sources (a help-center section, a policy page)
  • Resolvable in under four turns
  • A clear handoff target when the bot cannot help

Bad Week 1 use cases:

  • "Replace tier 1 support"
  • "Answer anything on the website"
  • "Handle billing, returns, and technical questions"

A narrow scope is not a humble scope. It is the only scope that finishes on time. For a step-by-step setup of a single pilot queue, see how to automate customer support.

Identify the 25-question golden test set

Pull 25 real questions from your support inbox that match the scope. Anonymize them. Include 10 common questions, 5 edge cases, 5 policy-adherence questions, 3 questions where the answer is genuinely not available (the bot should say so and offer a person), and 2 with a hostile or confused tone.

You will use this same test set every week for the rest of the pilot.

Pick the deployment surface

Decide where the bot will live in Week 3 before you build it. Options, from lower risk to higher risk:

  1. Internal-only test (your own team is the user)
  2. Help-center pages for the scoped topic only
  3. Specific product pages (for example, the returns page)
  4. A share of sessions across the site (needs session sampling in your own code if your platform has no percentage rollout)
  5. Main support page

Most pilots should pick option 2 or 3. The main support page is for after the pilot.

End-of-week-1 artifact: A one-page brief listing the use case, the three targets with their baselines, the stop rule, the 25-question test set and the pilot pages. Signed by the support lead and the exec sponsor.

Week 2: Build and offline test

Now you build. The constraint is that the bot has to pass the test set before it sees a real customer.

Build the bot

Configure the platform, add the approved knowledge and write the instructions. For a narrow use case this is a few days of work, not two weeks. A walkthrough of the build steps is in build your first AI agent, and the Chatsy specifics are in train your AI agent.

Resist the urge to add features. A pilot bot does not need 15 integrations. It needs to be correct on the use case you scoped.

Run the 25-question test set

Score every question on four dimensions: accuracy, tone, safety and handoff quality. (For the full method, see our chatbot evaluation guide.)

Set the pass criteria from your Week 1 targets, with one fixed rule: safety is 100%, and a bot that fails a safety question does not ship. For the other dimensions, write the threshold as a count. If you require 23 of 25 answers to be accurate, that is 23 ÷ 25 = 0.92, or 92%. Two wrong answers is the most you can accept.

If you do not pass, you iterate. You do not move to Week 3 with a failing offline score and hope production fixes it.

Set up handoff rules

Define when the bot hands off and to whom. These general rules work on most platforms:

  • Hand off on any explicit request ("I want to talk to a human")
  • Hand off when the bot cannot find a supporting source
  • Hand off when the visitor repeats the same question or says an answer did not help
  • Hand off on anything outside the scoped use case

In Chatsy, the Live chat module (Hobby plan and above) adds an "Escalate to Human" tool to the agent. Chatsy also offers a handoff or follow-up when the visitor asks for a person, repeats a question or says an answer did not help, when no supporting source is found, when the answer says it is unsure, or when an error occurs. Chatsy does not score sentiment, and it has no staff online or away status, so a handoff always goes to the queue. If nobody watches the queue during pilot hours, turn on email follow-up. Setup details are in the human handoff docs.

Make sure agents know the pilot is happening, what scope the bot covers and how to spot handoffs.

End-of-week-2 artifact: Test-set scores, the instructions and knowledge sources documented, the handoff rules written down, and an agent briefing complete.

Week 3: Soft launch

Week 3 puts the bot in front of real visitors on a small, controlled surface. The goal is to find the failures your test set missed while rollback is still easy.

Limit the launch to a small slice of traffic

Do not put the bot on your whole site on Day 1 of Week 3. Limit it to specific pages, or sample sessions in your own site code. Some platforms offer percentage rollouts or URL-targeting rules. Chatsy does not, so with Chatsy you add the embed script only to the pilot pages (install the widget).

The point of the limited rollout is that you can roll back instantly when something goes wrong. In Chatsy, turning off "Allow public chats" stops the embed script from loading, which works as a same-day kill switch. During Week 2 you can test on the private preview page (/preview/{agent ID}) before public chats are on.

Monitor real conversations daily

Every morning, read 20 to 30 conversations from the previous day. Look for:

  • Wrong answers (the bot said something inaccurate)
  • Bad handoffs (the bot handed off when it should have answered, or kept going when it should have handed off)
  • Tone misses (the bot was breezy with a frustrated customer)
  • Unscoped questions (customers asking about things outside the pilot scope)

The first week of real traffic always brings phrasings you did not test, typos, and questions that look like your scope but are actually different. Track every surprise in a doc. In Chatsy, the Conversations page and the Knowledge gaps list in Analytics are the two places to start.

Fix the top 3 failure modes

By Wednesday of Week 3, rank the failure modes by frequency and fix the top three. Do not try to fix all of them. The fixes are usually:

  • Adding missing knowledge content
  • Adjusting the instructions to handle a specific phrasing
  • Tightening or loosening a handoff rule

Re-run the 25-question test set after each fix to make sure you have not regressed.

End-of-week-3 artifact: A list of every conversation reviewed, the top failure modes ranked, the fixes shipped and the latest test-set scores.

Week 4: Measure and decide

This is the week the pilot exists for. Do not let it become "Week 3 part 2."

Compare actuals against criteria

On Monday of Week 4, pull the numbers:

  • Helpful share from thumbs ratings
  • Handoff rate
  • Wrong-answer rate from your conversation review
  • First reply time and in-scope ticket volume
  • Plan cost and usage for the month
  • CSAT, only if you ran your own survey

Put them next to the targets you wrote down in Week 1. Do not negotiate with the targets. The point of writing them down was to remove the temptation to move the goalposts.

Decide: scale, iterate, or kill

Three outcomes are possible.

Scale. You hit all three targets and did not break the stop rule. Write the rollout plan: which pages next, which scopes next, who owns it after the pilot. Get the rollout plan signed off the same week.

Iterate. You hit one or two targets and missed one by a small margin. Iterate is the most dangerous outcome because it has no natural end. Set a hard date (no more than four additional weeks) and a new gate. If you miss again, stop.

Kill. You missed two or three targets by clear margins, or you broke the stop rule. Turn the bot off the same day and write the post-mortem the same week.

The post-mortem is not punishment. It is the most valuable artifact of a failed pilot, because it tells the team what to scope differently next time.

End-of-week-4 artifact: A written decision (scale, iterate or kill) with the supporting numbers, the rollout or shutdown plan, and the next pilot scope if applicable.

Worked example: reading a 400-conversation pilot

Here is how the Week 4 decision works with real arithmetic. All inputs below are assumptions for illustration, not results from a Chatsy customer. Replace them with your own counts.

Input (assumed)Value
Conversations on the pilot pages in Week 3 and early Week 4400
Conversations with a thumbs rating120
Thumbs-up ratings90
Conversations handed to a person88
Conversations read in the review sample100
Wrong answers found in the review6
Safety failures0

The calculations, against the illustrative Week 1 targets:

  • Rating rate: 120 ÷ 400 = 0.30, so 30% of conversations were rated.
  • Helpful share: 90 ÷ 120 = 0.75, or 75%. Target at least 70%: met.
  • Handoff rate: 88 ÷ 400 = 0.22, or 22%. Target at most 30%: met.
  • Wrong-answer rate: 6 ÷ 100 = 0.06, or 6%. Target at most 5%: missed by 1 point.
  • Safety: 0 failures. Stop rule not broken.

Check how solid each number is before you act on it. With 120 ratings, one rating moves the helpful share by 100 ÷ 120 = 0.83 percentage points, so the 5-point margin over target equals 0.05 x 120 = 6 ratings. With 100 reviewed conversations, the 1-point miss is 0.01 x 100 = 1 conversation.

Decision: iterate. Two targets are met and the miss is one conversation in a small sample. Fix the sources behind the 6 wrong answers, re-run the 25-question test set, review another 100 conversations, and set a date two weeks out. If the wrong-answer rate is still above 5% on that date, stop the pilot.

Pilot vs production cost expectations

Price the pilot from your own volumes and the vendor's billing unit, not from a rule of thumb. Billing units differ: some vendors charge per resolution or per conversation, while Chatsy deducts credits for each AI response, from 1 to 30 credits per response depending on the model (message credits). Models on the Hobby and Standard plans cost at most 10 credits per response.

Using the same 400 conversations, assume 3 AI responses per conversation and 4 runs of the 25-question test set, since test chats are AI responses too:

  • Pilot responses: 400 x 3 = 1,200
  • Test-set responses: 25 x 4 = 100
  • Total: 1,200 + 100 = 1,300 AI responses in the month
Model cost per responseCredits neededHobby ($40, 500 credits, 1 seat)Standard ($150, 4,000 credits, 3 seats)Pro ($500, 15,000 credits, 5 seats)
1 credit1,300 x 1 = 1,300Not enoughFitsFits
10 credits1,300 x 10 = 13,000Not enoughNot enoughFits

The credit cap is strict: Chatsy blocks a response that would make usage reach the monthly allowance, so Hobby covers at most 499 one-credit responses. There are no top-ups, overages or rollover, and credits reset monthly. The Free plan ($0, 40 credits) has no live chat, so a pilot with human handoff needs Hobby or above. Seats include the owner, so only Standard and above let teammates work the handoff queue. See pricing for the full plan list.

On Standard with a 1-credit model, the plan price spread over the pilot is $150 ÷ 400 = $0.375 per pilot conversation. That is not your full cost. The 88 handed-off conversations still take agent time. Compare the total with your own fully loaded cost of handling the same 400 conversations by hand, using your payroll and ticket data. Support automation ROI walks through that calculation.

Not for you

Three kinds of pilots should not happen.

The vanity pilot. Leadership wants to say "we have AI." Nobody owns the metrics. The bot launches to a quiet corner of the site and nobody mentions it again. This is not a pilot, it is a press release. Kill it now and save the budget.

The delay tactic. "Let's pilot it for a quarter" usually means "let's not commit yet." If the pilot keeps getting extended without new evidence, the answer is not another extension. The answer is to call the question.

The pilot nobody can staff. If no one can read conversations daily in Week 3 or answer handoffs, the pilot will measure your staffing gap, not the bot. Wait until someone owns those hours.

A real pilot has a scope, a metric, a deadline and a kill switch. If yours does not, fix that before you spend another week on it.

FAQ

Q: Do 80% of AI projects really fail?

The figure is an estimate, not a measurement. RAND's 2024 report says that "by some estimates, more than 80 percent of AI projects fail," twice the rate of IT projects without AI, and cites other sources for that number. RAND's own interviews with 65 data scientists and engineers found that the most common root cause is a misunderstood or miscommunicated problem, followed by missing data. Treat the 80% as a warning about scope and data, not as a forecast for your pilot.

Q: Four weeks seems short. Can we do an 8-week pilot?

Yes, but the structure is the same. Add the weeks to Week 2 (build) or Week 3 (soft launch), not to Week 1 or Week 4. Longer scoping rarely improves outcomes, and longer measurement rarely changes the decision.

Q: What if our success criteria turn out to be wrong?

Acknowledge it in writing during Week 4. For example: "We targeted 75% helpful and reached 68%, and every unhelpful rating was about exchanges, which our help center does not cover." Then run a second pilot against the new criteria. Do not silently move the targets mid-pilot.

Q: Who owns the pilot?

One named person, full stop. Usually a support lead, with engineering and product as supporting players. If three people share ownership, nobody owns it.

Run the four weeks and get an answer

A pilot is not a marketing exercise. It is a structured way to find out whether a chatbot is worth scaling at your company. Done well, it takes four weeks and produces a clear yes or no. Done poorly, it produces a permanent half-deployment that costs the team trust.

If you run the pilot in Chatsy, you can train the agent from web pages, files or written answers, test it privately on the preview page, add the embed script only to your pilot pages, turn on human handoff with the Live chat module on Hobby and above, and track conversations, helpful ratings, handoff rate and knowledge gaps. Chatsy does not offer percentage rollouts, URL-targeting rules, a domain allowlist, CSAT surveys or containment reports, so plan those steps in your own site code or tools. Start free to test answers, or see pricing for the plans that include handoff.

All articles
  • AI & Technology

    AI Chatbot Evaluation and Testing: A Support Team Playbook for 2026

    How to evaluate a customer support chatbot before launch using a 25-question golden set, four scoring dimensions, and the right tooling stack.

    May 20, 2026 · 12 min read

Build an agent from your support content

Add approved help content, test its answers, and decide when a person takes over.

No credit card required