# What is a RAG chatbot? How knowledge base chatbots work

A RAG chatbot is a chatbot that looks up relevant passages in your own documents before it answers, then writes the answer from those passages. RAG stands for retrieval-augmented generation. It is the standard way to build a knowledge base chatbot, because the answers come from content you control and can be traced to a source, and updating the bot means updating a document, not retraining a model.

By Shai Goldenberg. Updated: 2026-11-14.

## The problem RAG solves

A language model knows what was in its training data. It does not know your return policy, your product manual or the pricing you changed last week. Ask it anyway and it may produce a fluent answer that is wrong.

Retrieval-augmented generation fixes the knowledge gap without changing the model. Before the model answers, the system searches your content for passages related to the question and puts them in front of the model with an instruction: answer from this material, and say so if the answer is not there.

A knowledge base chatbot and a RAG chatbot are, in practice, the same thing. "Knowledge base chatbot" names what it does. "RAG" names how.

## How a RAG chatbot works, step by step

There are two phases. One runs ahead of time, one runs on every question.

### Phase 1: indexing your content

1. **Collect.** Pull the documents from where they live: help center, PDFs, wiki, policy files, product data.
2. **Clean.** Remove navigation, boilerplate and duplicates. Keep titles, headings and dates, because they help retrieval later.
3. **Chunk.** Split each document into passages. A passage should be small enough to be specific and large enough to make sense alone. Splitting along headings works better than cutting every fixed number of characters.
4. **Embed.** An embedding model turns each passage into a list of numbers that represents its meaning. Passages about similar things get similar numbers.
5. **Store.** Save the passages, their embeddings and their metadata (source, date, who may see it) in a search index.

### Phase 2: answering a question

1. **Understand the question.** In a conversation, "and what about for business customers" only makes sense with the previous message. The system rewrites the question into a standalone query.
2. **Retrieve.** Search the index for the passages closest in meaning to the query. Good systems combine meaning-based search with keyword search, because exact terms such as model numbers and error codes do not embed well.
3. **Filter by permission.** Remove anything the person asking is not allowed to see.
4. **Rerank.** Sort the candidates so the most relevant few go to the model.
5. **Generate.** The model receives the question, the passages and the instructions, and writes the answer.
6. **Cite.** The answer links to the passages it used.

Embeddings in plain words: they let the system match "how do I send something back" to a document titled "Returns policy", although the two share no words.

## What decides the quality

Teams often spend their effort choosing a model. In a RAG system, the model is rarely the weak point. These are.

**The content.** A chatbot cannot be more correct than its sources. Outdated pages, two documents that contradict each other, and knowledge that exists only in people's heads all show up as wrong answers. A content cleanup is usually part of the project.

**Chunking.** A passage cut in the middle of a table or separated from its heading loses its meaning. This is unglamorous work and it moves answer quality more than most other choices.

**Retrieval.** If the right passage is not in the handful sent to the model, the answer will be wrong or empty no matter how capable the model is. Measure retrieval separately from generation.

**Instructions.** The model must be told to answer only from the sources, to say when they do not cover the question, and what to do then: ask a clarifying question or hand off to a person.

## Permissions: the part enterprises get wrong

In a public help center bot, everyone may see everything. Inside a company, that is false. HR documents, contracts, finance reports and customer records have different audiences.

The rule is simple to state: the chatbot must never show a person content that person could not open directly. To enforce it, each passage carries the access rules of its source document, and retrieval filters by the identity of the person asking, on the server, before anything reaches the model. Telling the model "do not reveal confidential information" in a prompt is not access control.

Three related points for an [enterprise chatbot](https://goldenroz.com/enterprise-chatbot-development/):

- **Sync permissions, not only content.** When someone loses access to a folder, the bot must stop answering from it.
- **Log every question** with the user, the passages retrieved and the answer, so an incident can be investigated.
- **Treat documents as untrusted input.** A document can contain text written to manipulate the model, known as prompt injection. Retrieved text is data, never instructions.

Shai's on-prem MCP gateway applies the same principles to structured systems: per-user permissions, PII masking, an audit log and approved queries only. For questions that need live data from an ERP or CRM, a connector of that kind works alongside document retrieval. The background is in [what MCP is](https://goldenroz.com/blog/what-is-mcp/), and the service is [AI integration](https://goldenroz.com/ai-integration/).

## Citations

A citation is a link from a sentence in the answer to the passage it came from. Citations do three jobs. The user can check the answer. The support team can see which document caused a wrong answer and fix it. And an answer without a supporting source becomes visible as a warning sign.

For internal bots, citations are close to mandatory. People act on the answers, and "the bot said so" is not a source.

## Evaluation: how to know it works

A RAG chatbot is judged on a test set, not on a demo. Build a list of real questions with the correct answer and the document that contains it. Then measure:

| Metric | Question it answers |
|---|---|
| Retrieval hit rate | Was the right passage among those retrieved? |
| Groundedness | Is every claim in the answer supported by the retrieved passages? |
| Correctness | Does the answer match the expected answer? |
| Abstention | When the sources do not cover the question, does the bot say so? |
| Citation accuracy | Do the cited passages support the sentence they are attached to? |

Run the set on every change to content, chunking, prompts or model. Add each real failure from production to the set. Without this, every improvement is a guess and every model upgrade is a risk.

## Failure modes to plan for

- **The answer is not in the knowledge base.** The bot should say so and offer a handoff. A bot that always answers is a bot that sometimes invents.
- **Stale content.** The index must update when documents change, and removed documents must disappear from it.
- **Conflicting sources.** Two policies, two answers. Prefer the newer or the authoritative source through metadata, and fix the content.
- **Questions that need a calculation or live data.** "What is my order status" is not in any document. That needs a tool call to a system, not retrieval.
- **Questions across many documents.** "Summarize all complaints this quarter" is an analysis task. Retrieval of a few passages cannot answer it.
- **Tables and scanned PDFs.** These need extraction work before they are searchable.

## RAG vs fine-tuning vs long context

| | RAG | Fine-tuning | Long context |
|---|---|---|---|
| How knowledge gets in | Retrieved per question | Trained into the model | Whole documents placed in the prompt |
| Updating knowledge | Update the document and re-index | Retrain | Replace the documents |
| Per-user permissions | Yes, by filtering retrieval | No | Only by choosing documents per user |
| Citations | Natural | Not available | Possible |
| Scales to large content | Yes | Not suited to facts that change | Limited by the context size and cost per question |
| Best for | Company knowledge that changes | Style, format, narrow tasks | Small, stable document sets |

Long context deserves a note. Current models accept very large inputs, so for a few dozen pages you can skip retrieval and include everything, often with prompt caching to control cost. It is the simplest design and worth trying first when the content is small. Retrieval becomes necessary when the content is large, changes often or has per-user permissions.

Fine-tuning is rarely the answer to "the model does not know our content". It fits when you need a consistent output format or tone that instructions alone do not achieve.

## What a build includes

A realistic scope for [RAG chatbot development](https://goldenroz.com/rag-chatbot-development/) has six parts: connectors to the content sources, the indexing pipeline, retrieval with permission filtering, the chat interface on your chosen channel, the evaluation set, and monitoring with a feedback loop. The same components serve a support bot, covered in our [customer service chatbot guide](https://goldenroz.com/blog/customer-service-chatbot-guide/).

We describe this as how we build it. We do not publish client results for RAG projects, and you should be skeptical of anyone who quotes an accuracy figure before seeing your content.

## What to do next

Collect 50 real questions your team or customers asked last month, and for each one find the document that answers it. The questions with no document show your content gaps. The rest become your first evaluation set. With that in hand, a scoping call is short and specific. See [RAG chatbot development](https://goldenroz.com/rag-chatbot-development/) for the service. Projects start at $25,000 at a fixed price per phase, and the first one-hour consultation is free.

## Questions buyers ask

### What does RAG stand for?

RAG stands for retrieval-augmented generation. Retrieval means finding the passages in your content that relate to the question. Generation means the language model writes an answer using those passages. The model is augmented with your content at the moment of the question.

### What is the difference between a RAG chatbot and a normal LLM chatbot?

A plain LLM chatbot answers from what the model learned in training, which does not include your internal documents and may be out of date. A RAG chatbot is given the relevant parts of your content with each question and is instructed to answer from them. That makes its answers specific to your business and checkable against a source.

### Does a RAG chatbot stop hallucinations?

It reduces them and makes them easier to catch, but it does not remove them. If retrieval returns the wrong passage, or no passage, the model can still produce a confident wrong answer. Citations, an instruction to say when the answer is not in the sources, and a regular evaluation set are the controls.

### Is RAG better than fine-tuning?

For answering from company knowledge, usually yes. RAG uses current documents, respects permissions per user and shows sources. Fine-tuning changes how a model behaves or writes, and is a poor way to store facts that change. The two can be combined, but most knowledge base chatbots need only RAG.

### Do I need a vector database for a RAG chatbot?

Not always. Vector search finds passages by meaning, which helps when users phrase things differently from the documents. Keyword search is better for exact terms such as part numbers. Many systems combine both. For a small, stable knowledge base, putting the documents directly in the model's context can be enough.

---
Page: https://goldenroz.com/blog/what-is-a-rag-chatbot/
Book a free one-hour call: https://calendly.com/daniarozin/daniel-rozin-consultation-1-hour-clone
