Build a Free Llama 3 Chatbot That Answers From Your Own Files

2026-11-08

A free Llama 3 chatbot needs three things on one machine: the Ollama runtime, one chat model, one embedding model. No API key, no per-token bill. What you pay in is disk space, memory and patience, so the first decision is which model size your hardware can carry.

The goal here is a small script that reads your .txt files, finds the passages closest to a question and hands them to Llama 3.1 to answer. It is the same retrieve-then-answer design as in the PDF version of this build, only with the models running locally.

What to pick: model size

Ollama publishes the download size of every tag. As of November 2026 the llama3.1 page lists these, all with a 128K context window.

TagDownload sizeContext
llama3.1:8b (same as latest)4.9 GB128K
llama3.1:70b43 GB128K
llama3.1:405b243 GB128K
llama3:8b (older Llama 3)4.7 GB8K
llama3:70b (older Llama 3)40 GB8K

Source: Ollama library, llama3.1; the two Llama 3 rows come from the Ollama library page for llama3.

8b4.9 GB70b43 GB405b243 GB
Download size by Llama 3.1 tag, in gigabytes

The pages give download size, not RAM. Loaded weights take at least that much, and the context window adds more on top, so the 8B tag is the realistic choice for a laptop and the 70B tag is a workstation job. Older llama3 stops at 8K of context; llama3.1 is the one to pull for a document bot.

What you need

  • Ollama installed (on Linux and macOS: curl -fsSL https://ollama.com/install.sh | sh)
  • Python 3 with requests
  • a folder docs/ with plain .txt files

Pull both models:

ollama pull llama3.1
ollama pull nomic-embed-text

nomic-embed-text is about 274 MB and accepts 2K tokens per input, according to its Ollama page, which matters for the chunking below. Ollama listens on http://localhost:11434 by default.

1Embed thequestion withnomic-embed-text2Compare it withthe stored chunkvectors3Take the bestfew chunks4Send chunks andquestion tollama3.15Print the answerwith its sourcefile
How a question travels through the local bot

Step 1: split files into chunks

Cut each file on blank lines and glue paragraphs until a chunk is about 1,200 characters. That stays far under the 2K-token limit of the embedding model. A typical mistake is splitting by a fixed character count, which cuts sentences in half and makes the matching worse. Chunking is covered in more depth in the PDF guide linked above.

Step 2: embed once, keep the vectors

The /api/embed endpoint takes a model and an input that can be a list of strings, and returns an embeddings array (Ollama API reference). Embed all chunks in one call and keep them in memory.

import math
import pathlib
import requests

OLLAMA = "http://localhost:11434"

def embed(texts):
    r = requests.post(f"{OLLAMA}/api/embed",
                      json={"model": "nomic-embed-text", "input": texts})
    r.raise_for_status()
    return r.json()["embeddings"]

def load_chunks(folder="docs", size=1200):
    chunks = []
    for path in pathlib.Path(folder).glob("*.txt"):
        buf = ""
        for para in path.read_text(encoding="utf-8").split("\n\n"):
            if len(buf) + len(para) > size and buf:
                chunks.append((path.name, buf.strip()))
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            chunks.append((path.name, buf.strip()))
    return chunks

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

CHUNKS = load_chunks()
VECTORS = embed([text for _, text in CHUNKS])

For a few hundred chunks plain Python is enough. A mistake that makes results look random: embedding the documents with one model and the question with another. Use nomic-embed-text for both.

Step 3: retrieve and ask

The /api/chat endpoint takes model, messages and stream. With "stream": False the reply arrives as one object, and the text sits in message.content.

def answer(question, k=3):
    q = embed([question])[0]
    ranked = sorted(range(len(CHUNKS)),
                    key=lambda i: cosine(q, VECTORS[i]), reverse=True)[:k]
    context = "\n\n".join(f"[{CHUNKS[i][0]}]\n{CHUNKS[i][1]}" for i in ranked)
    messages = [
        {"role": "system", "content": "Answer only from the context. "
         "If the context does not contain the answer, say so. "
         "Name the file in brackets that you used."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ]
    r = requests.post(f"{OLLAMA}/api/chat",
                      json={"model": "llama3.1", "messages": messages, "stream": False})
    r.raise_for_status()
    return r.json()["message"]["content"]

print(answer("What is the notice period in the contract template?"))

Two Ollama defaults decide how this behaves. The FAQ says it "uses a context window size of 4096 tokens" unless OLLAMA_CONTEXT_LENGTH is set, so three chunks of 1,200 characters fit comfortably but twenty will not: the oldest text is cut silently. The same FAQ says models stay in memory for 5 minutes after the last request, so the first question after a pause is slower while the model reloads.

Step 4: check that it answers from your files

Ask three questions and read each answer against the source file:

  1. one whose answer sits in a single chunk;
  2. one whose answer is in no file, where the right reply is that the context does not contain it;
  3. one whose answer is spread over two chunks, which shows whether k=3 is enough.

If the second one gets a confident answer, tighten the system prompt before touching anything else. A small local model follows the "say so" rule less reliably than a large hosted one, and no setting turns that into a guarantee. For a question that needs several searches, see RAG chatbot vs agentic AI.

Where this stops

Ollama binds to 127.0.0.1 by default, so this bot is reachable from your own machine only. Putting it behind a website means a server that is always on and an access layer in front, which is a separate job. And the 8B model is a modest reader: when answers come out thin, raise k or improve the chunks first, then consider the 70B tag.