GoldenrozBook a call

Customer service chatbots: what to automate and what to keep human

A customer service chatbot should automate requests that are frequent, have a clear correct answer and can be resolved from your knowledge base or a lookup in your systems, such as order status, policies and how-to questions. Complaints, disputes, exceptions and anything emotional or high-stakes should go to a person with the full context attached. Measure the bot by problems actually resolved, not by conversations that ended without a human.

A gold sphere connected by thin rods to six small navy spheres.

Start with the tickets, not the technology

The decision about what to automate is made in your ticket history. Export the last few months of support requests and group them by reason. For most support teams a small number of reasons accounts for a large share of volume. Those reasons, not a feature list, define the project.

For each group, ask three things:

  1. Is there one correct answer? “Where is my order” has one. “I am unhappy with the product” does not.
  2. Where does the answer live? In a help article, in a system, or in an experienced agent’s head.
  3. What does a wrong answer cost? A wrong opening-hours answer is an annoyance. A wrong refund promise is money and trust.

Which ticket types suit a bot

Ticket type Automate? What the bot needs
How-to and product questions Yes A current knowledge base
Policies: shipping, returns, warranty Yes The policy documents, with dates
Order status and tracking Yes Read access to the order system
Appointment booking and changes Yes Calendar access with rules
Address or contact detail changes Yes, with verification Write access, identity check
Refunds and returns within policy Partly Rules, limits, and approval above a threshold
Troubleshooting with a known decision path Partly Structured steps, handoff when they run out
Billing disputes No. Collect details, then hand off A good intake form
Complaints and upset customers No Fast routing to a person
Exceptions to policy No Human judgment
Safety, legal or regulatory issues No A fixed rule that routes immediately

“Partly” is the important category. In these cases the bot does the first part, gathering the order number, the reason and the photos, and either finishes the case under a rule or hands a complete file to an agent. That saves agent time even when the bot does not close the ticket.

Grounding: where the answers come from

A support bot is only as correct as its sources. There are two kinds, and most bots need both.

The knowledge base

For general questions, the bot retrieves the relevant passages from your help center and policy documents and answers from them, with a link to the source. This is retrieval-augmented generation, explained in what a RAG chatbot is. Our RAG chatbot development page describes the build.

Three rules:

  • The bot answers only from the sources. If they do not cover the question, it says so and offers a person.
  • Fix the content first. Contradicting articles produce contradicting answers.
  • Internal notes and macros written for agents are not automatically safe to show customers. Decide which sources the bot may use.

Customer and order data

“Where is my order” is not in any document. The bot needs a tool that queries the order system for this customer. That requires:

  • Identity verification before showing anything personal. On a logged-in website this comes from the session. On WhatsApp or the phone, verify with something the customer knows or a one-time code.
  • Scoped access. The tool returns this customer’s records only. The limit is enforced in your code, not in the prompt.
  • Read first, write later. Start with lookups. Add actions one at a time, each with limits.

When the bot starts issuing refunds or changing records by itself, it is an agent, and the permission, approval and audit work grows. That stage is covered by our customer service AI agent page, and the distinction in AI agent vs chatbot.

Escalation: the part customers remember

Customers forgive a bot that cannot help. They do not forgive one that will not let them reach a person.

When to escalate

  • The customer asks for a person. Always honor it, the first time.
  • The bot failed to resolve after two attempts.
  • The topic is on the always-human list.
  • The message shows anger or distress.
  • The request needs an action above the bot’s limit.

How to escalate

  • Pass the full conversation, the verified identity, and a short summary with what was already tried.
  • Tell the customer what happens next and how long it is likely to take.
  • Outside working hours, say so, create the ticket and confirm it.
  • Keep the bot silent while the agent has the conversation.

An escalation with full context is a success for the bot, not a failure. The agent starts at minute three instead of minute zero.

Measuring containment honestly

Containment rate, the share of conversations that never reach a human, is the number every vendor quotes. On its own it rewards the wrong behavior. A bot that hides the “talk to a person” option has excellent containment and terrible service.

A conversation can end without a human for four different reasons:

  1. The problem was solved.
  2. The customer gave up.
  3. The customer went to another channel, such as phone or email.
  4. The customer got a wrong answer and believed it.

Only the first is a win. To separate them, measure a set of numbers together.

Metric What it tells you Watch for
Containment rate Share of conversations with no handoff High containment with falling satisfaction
Confirmed resolution Customer said the issue was solved, or completed the task Low response rates to the question
Repeat contact rate Same customer, same topic, within a few days on any channel Rises when the bot gives non-answers
Escalation quality Share of handoffs where the agent had what was needed Agents re-asking for details
Answer accuracy Sampled answers checked against the sources by a person The only direct check on wrong answers
Satisfaction after bot conversations Compared with human-handled conversations of the same type Compare like with like

Two practices keep the numbers honest. First, a person reviews a random sample of bot conversations every week and labels them resolved, abandoned or wrong. Second, repeat contacts are tracked across channels, so a customer who leaves the chat and calls is counted.

We do not publish a “typical” containment figure. It depends on your ticket mix, and a number borrowed from another company tells you nothing about yours. Set the baseline from your own first month.

A rollout plan

Phase 1: answers only. The bot answers from the knowledge base on one channel, with handoff. No customer data. Goal: learn what people actually ask, and build the evaluation set from real conversations.

Phase 2: lookups. Add identity verification and read access to orders or accounts. The bot now answers the highest-volume personal questions.

Phase 3: intake for humans. For the always-human categories, the bot collects the details and files a complete ticket.

Phase 4: limited actions. Add one action at a time, starting with the cheapest to reverse, each with a limit and an approval rule above it.

At every phase, launch to a share of traffic first, review transcripts daily for the first two weeks, and widen only when accuracy and escalation quality hold.

Shai’s work includes a support assistant with a knowledge base and a production WhatsApp ordering bot on Claude, built with model fallbacks, timeouts, retries and deterministic fallback replies. The same failure handling applies to a support bot: when the model is unavailable, the customer still gets a reply and a route to a person.

Enterprise requirements

A large organization adds requirements that do not change the conversation and do change the build:

  • Single sign-on for agents and admins, and for internal users of an employee-facing bot.
  • Per-user permissions, so the bot never shows content the asker could not open.
  • Audit logs of every question, source and action.
  • Data retention rules and a clear answer on what is sent to the model provider.
  • Integration with the existing help desk, so there is one ticket history.
  • A change process: prompts, sources and tools are versioned and tested before release.

What to keep human, on purpose

Some conversations are worth more with a person even when a bot could handle them. A customer about to cancel, a first-time buyer of an expensive product, a long-standing account with a problem. Automating the routine requests is what gives your team time for those. That is the purpose of the project. Removing people from support is not.

Pre-launch checklist

  • Ticket reasons ranked by volume, each marked automate, partly or human
  • Knowledge base reviewed for outdated and conflicting articles
  • Always-human topic list agreed with the support lead
  • Identity verification defined per channel
  • Handoff passes conversation, identity and summary into the help desk
  • Evaluation set of real questions with expected answers
  • Metrics beyond containment set up, including repeat contact
  • Weekly transcript review assigned to a named person

What to do next

Export three months of tickets, group them by reason and apply the table above. The top automatable reasons are the scope of phase 1. Our customer service chatbot service builds it on your channels and help desk. For budgeting, see how much a chatbot costs. Projects start at $25,000 at a fixed price per phase, and the first one-hour consultation is free.

Questions buyers ask

What can a customer service chatbot handle?

Questions answered by your help center and policies, status lookups such as order tracking, simple account changes, and collecting details before a handoff. These are high-volume requests with one correct answer. A bot connected to your order system can also carry out limited actions under set rules.

What should not be automated in customer support?

Complaints, billing disputes, cancellations where you want a retention conversation, safety issues, legal threats, and any case that needs an exception to policy. These need judgment and empathy, and a wrong automated reply costs more than the time saved.

What is containment rate?

Containment rate is the share of conversations that end without being passed to a human. It is easy to measure and easy to misread, because a customer who gives up counts as contained. Pair it with resolution checks, repeat contact rate and satisfaction scores.

How does a support chatbot know the right answer?

It is grounded in your sources. For general questions it retrieves passages from the knowledge base and answers from them. For customer-specific questions it calls your order, billing or account system through a tool. A bot that answers from the model's general knowledge will invent policies.

What is an enterprise chatbot?

An enterprise chatbot is one deployed in a large organization with the controls that requires: single sign-on, per-user permissions on what it can show, audit logs, integration with systems of record and data handling that meets company policy. The conversation is the same. The access control and governance are what differ.

Written by

CTO and technology partner

Shai leads the technology side: business software, SaaS and mobile apps, payments and escrow systems, Postgres security, AI agents built with Claude Code and MCP, and IT and infrastructure projects with senior infrastructure engineers.

Keep reading

Talk to the person who will do the work

One hour, no sales team. You talk to Daniel or Shai, we look at your account or your product, and you leave with a clear next step.

Or send the details

Book a 1-hour consultationWhatsApp us