RAG Chatbot vs Agentic AI in Practice
2026-11-01
Short answer to "is a RAG chatbot agentic AI": no, not by default. A classic RAG chatbot follows a path you wrote. Retrieve, put the passages into a prompt, answer. An agentic one is handed tools and decides which to call, in what order, and when it has enough. The difference is who controls the loop.
What you need to follow along: a search function over your documents (a vector store or a keyword index) and a chat model that supports tool calling. The code uses two placeholders, search(query, k) returning a list of text passages and llm(messages, tools=None) returning a reply. The call counts below come from the design of the code, not from a benchmark.
The task
A support bot over a company's own documents gets this question: "Is an annual plan bought through a reseller refundable?" Suppose the answer sits in two files. The refund policy gives annual plans a refund window, and the reseller terms say resold plans are handled by the reseller. No single passage holds the whole answer.
Fixed RAG in four steps
- Embed the question. Turn the user text into the same kind of vector or keyword query the index uses. Typical mistake: indexing documents with one embedding model and querying with another, so results look random.
- Retrieve the top passages. Take the best
k, here 4. Typical mistake:ktoo small, so the second file never appears, and nothing in the pipeline notices. - Build the prompt. Passages in, instruction "answer only from the context". Typical mistake: leaving out the "say so if the context does not contain the answer" rule, which makes the bot guess.
- Answer. One model call. Typical mistake: showing the answer without the source passage, so nobody can see which file it came from.
def answer_rag(question: str) -> str:
passages = search(question, k=4)
context = "\n\n".join(passages)
messages = [
{"role": "system", "content": "Answer only from the context. "
"If the context does not contain the answer, say so."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
]
return llm(messages)
One retrieval, one model call, the same cost for every question. For chunking and a test for questions the documents cannot answer, build that base first; the loop below only adds to it.
The same task as a tool loop
- Describe the tool. Give the model
search_docswith a name, a description and aqueryparameter. Typical mistake: a vague description, so the model never calls it or calls it for everything. - Let the model choose. Each turn it either calls the tool or writes the answer. Typical mistake: forgetting to append the tool result to the messages, so the model asks for the same search again.
- Run the call and feed the result back. Typical mistake: returning raw, very long text, which fills the context in two rounds.
- Stop. On a plain reply, or at a hard step limit. Typical mistake: no limit.
TOOLS = [{
"type": "function",
"function": {
"name": "search_docs",
"description": "Search the company documents. Returns up to 4 passages.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
}]
def answer_agent(question: str, max_steps: int = 4) -> str:
messages = [
{"role": "system", "content": "Use search_docs as many times as needed. "
"Answer only from what it returns. If it does not return the answer, say so."},
{"role": "user", "content": question},
]
for _ in range(max_steps):
reply = llm(messages, tools=TOOLS)
if not reply.tool_calls:
return reply.content
messages.append(reply.as_message())
for call in reply.tool_calls:
result = search(call.arguments["query"], k=4)
messages.append({"role": "tool", "tool_call_id": call.id,
"content": "\n\n".join(result)})
return "Could not find a complete answer within the step limit."
Message field names differ between providers, so read this as a sketch and check your SDK.
Counting the calls
For the refund question a plausible run is: call 1, the model asks for "refund policy annual plan"; call 2, it sees the policy and asks for "reseller terms refund"; call 3, it has both and writes the answer. Three model calls and two searches, against one and one for the fixed build. With max_steps=4 the worst case is four model calls and four searches; without a cap there is no upper bound, which is why Neo4j's explainer recommends a hard limit on loop iterations.
Calls are only part of the cost, because each pass resends the system prompt, the tool description and everything gathered so far. A 2026 preprint by Ferrazzi, Cvjeticanin, Piraccini and Giannuzzi compared agentic RAG with an enhanced, pipeline-based RAG. It reports that the agentic settings needed on average 3.3 times more input tokens, 1.9 times more output tokens and 1.5 times more time across the datasets it tested (arXiv 2601.07711). Those are results on their datasets and models, not a forecast for your bot.
Side by side
| Fixed RAG | Tool loop | |
|---|---|---|
| Who picks the next step | Your code | The model |
| Model calls per question | 1 | 1 to max_steps |
| Searches per question | 1 | 0 to max_steps |
| Question that needs two documents | Works only if both rank in the top k | Can search again with a new query |
| Debugging | Read one prompt | Read a transcript |
| Main risk | Silent miss on a split answer | Cost growth and loops |
Often the cheaper fix for a split answer is no agent at all: retrieve more passages, rewrite the question before searching, or add the missing link between the two documents to the source text. The Neo4j article also describes an adaptive pattern, where simple questions go straight through and only hard ones enter the loop.