Customer service chatbots: what to automate and what to keep human
A customer service chatbot should automate requests that are frequent, have a clear correct answer and can be resolved from your knowledge base or a lookup in your systems, such as order status, policies and how-to questions. Complaints, disputes, exceptions and anything emotional or high-stakes should go to a person with the full context attached. Measure the bot by problems actually resolved, not by conversations that ended without a human.

Start with the tickets, not the technology
The decision about what to automate is made in your ticket history. Export the last few months of support requests and group them by reason. For most support teams a small number of reasons accounts for a large share of volume. Those reasons, not a feature list, define the project.
For each group, ask three things:
- Is there one correct answer? “Where is my order” has one. “I am unhappy with the product” does not.
- Where does the answer live? In a help article, in a system, or in an experienced agent’s head.
- What does a wrong answer cost? A wrong opening-hours answer is an annoyance. A wrong refund promise is money and trust.
Which ticket types suit a bot
| Ticket type | Automate? | What the bot needs |
|---|---|---|
| How-to and product questions | Yes | A current knowledge base |
| Policies: shipping, returns, warranty | Yes | The policy documents, with dates |
| Order status and tracking | Yes | Read access to the order system |
| Appointment booking and changes | Yes | Calendar access with rules |
| Address or contact detail changes | Yes, with verification | Write access, identity check |
| Refunds and returns within policy | Partly | Rules, limits, and approval above a threshold |
| Troubleshooting with a known decision path | Partly | Structured steps, handoff when they run out |
| Billing disputes | No. Collect details, then hand off | A good intake form |
| Complaints and upset customers | No | Fast routing to a person |
| Exceptions to policy | No | Human judgment |
| Safety, legal or regulatory issues | No | A fixed rule that routes immediately |
“Partly” is the important category. In these cases the bot does the first part, gathering the order number, the reason and the photos, and either finishes the case under a rule or hands a complete file to an agent. That saves agent time even when the bot does not close the ticket.
Grounding: where the answers come from
A support bot is only as correct as its sources. There are two kinds, and most bots need both.
The knowledge base
For general questions, the bot retrieves the relevant passages from your help center and policy documents and answers from them, with a link to the source. This is retrieval-augmented generation, explained in what a RAG chatbot is. Our RAG chatbot development page describes the build.
Three rules:
- The bot answers only from the sources. If they do not cover the question, it says so and offers a person.
- Fix the content first. Contradicting articles produce contradicting answers.
- Internal notes and macros written for agents are not automatically safe to show customers. Decide which sources the bot may use.
Customer and order data
“Where is my order” is not in any document. The bot needs a tool that queries the order system for this customer. That requires:
- Identity verification before showing anything personal. On a logged-in website this comes from the session. On WhatsApp or the phone, verify with something the customer knows or a one-time code.
- Scoped access. The tool returns this customer’s records only. The limit is enforced in your code, not in the prompt.
- Read first, write later. Start with lookups. Add actions one at a time, each with limits.
When the bot starts issuing refunds or changing records by itself, it is an agent, and the permission, approval and audit work grows. That stage is covered by our customer service AI agent page, and the distinction in AI agent vs chatbot.
Escalation: the part customers remember
Customers forgive a bot that cannot help. They do not forgive one that will not let them reach a person.
When to escalate
- The customer asks for a person. Always honor it, the first time.
- The bot failed to resolve after two attempts.
- The topic is on the always-human list.
- The message shows anger or distress.
- The request needs an action above the bot’s limit.
How to escalate
- Pass the full conversation, the verified identity, and a short summary with what was already tried.
- Tell the customer what happens next and how long it is likely to take.
- Outside working hours, say so, create the ticket and confirm it.
- Keep the bot silent while the agent has the conversation.
An escalation with full context is a success for the bot, not a failure. The agent starts at minute three instead of minute zero.
Measuring containment honestly
Containment rate, the share of conversations that never reach a human, is the number every vendor quotes. On its own it rewards the wrong behavior. A bot that hides the “talk to a person” option has excellent containment and terrible service.
A conversation can end without a human for four different reasons:
- The problem was solved.
- The customer gave up.
- The customer went to another channel, such as phone or email.
- The customer got a wrong answer and believed it.
Only the first is a win. To separate them, measure a set of numbers together.
| Metric | What it tells you | Watch for |
|---|---|---|
| Containment rate | Share of conversations with no handoff | High containment with falling satisfaction |
| Confirmed resolution | Customer said the issue was solved, or completed the task | Low response rates to the question |
| Repeat contact rate | Same customer, same topic, within a few days on any channel | Rises when the bot gives non-answers |
| Escalation quality | Share of handoffs where the agent had what was needed | Agents re-asking for details |
| Answer accuracy | Sampled answers checked against the sources by a person | The only direct check on wrong answers |
| Satisfaction after bot conversations | Compared with human-handled conversations of the same type | Compare like with like |
Two practices keep the numbers honest. First, a person reviews a random sample of bot conversations every week and labels them resolved, abandoned or wrong. Second, repeat contacts are tracked across channels, so a customer who leaves the chat and calls is counted.
We do not publish a “typical” containment figure. It depends on your ticket mix, and a number borrowed from another company tells you nothing about yours. Set the baseline from your own first month.
A rollout plan
Phase 1: answers only. The bot answers from the knowledge base on one channel, with handoff. No customer data. Goal: learn what people actually ask, and build the evaluation set from real conversations.
Phase 2: lookups. Add identity verification and read access to orders or accounts. The bot now answers the highest-volume personal questions.
Phase 3: intake for humans. For the always-human categories, the bot collects the details and files a complete ticket.
Phase 4: limited actions. Add one action at a time, starting with the cheapest to reverse, each with a limit and an approval rule above it.
At every phase, launch to a share of traffic first, review transcripts daily for the first two weeks, and widen only when accuracy and escalation quality hold.
Shai’s work includes a support assistant with a knowledge base and a production WhatsApp ordering bot on Claude, built with model fallbacks, timeouts, retries and deterministic fallback replies. The same failure handling applies to a support bot: when the model is unavailable, the customer still gets a reply and a route to a person.
Enterprise requirements
A large organization adds requirements that do not change the conversation and do change the build:
- Single sign-on for agents and admins, and for internal users of an employee-facing bot.
- Per-user permissions, so the bot never shows content the asker could not open.
- Audit logs of every question, source and action.
- Data retention rules and a clear answer on what is sent to the model provider.
- Integration with the existing help desk, so there is one ticket history.
- A change process: prompts, sources and tools are versioned and tested before release.
What to keep human, on purpose
Some conversations are worth more with a person even when a bot could handle them. A customer about to cancel, a first-time buyer of an expensive product, a long-standing account with a problem. Automating the routine requests is what gives your team time for those. That is the purpose of the project. Removing people from support is not.
Pre-launch checklist
- Ticket reasons ranked by volume, each marked automate, partly or human
- Knowledge base reviewed for outdated and conflicting articles
- Always-human topic list agreed with the support lead
- Identity verification defined per channel
- Handoff passes conversation, identity and summary into the help desk
- Evaluation set of real questions with expected answers
- Metrics beyond containment set up, including repeat contact
- Weekly transcript review assigned to a named person
What to do next
Export three months of tickets, group them by reason and apply the table above. The top automatable reasons are the scope of phase 1. Our customer service chatbot service builds it on your channels and help desk. For budgeting, see how much a chatbot costs. Projects start at $25,000 at a fixed price per phase, and the first one-hour consultation is free.