Local RAG With Open WebUI: Chat With Documents Without the Cloud

Uploading a PDF to a local chatbot feels impressive right up to the moment it confidently invents a clause that is not in the file. RAG does not make a language model truthful. It gives the model a search system and some retrieved evidence. Extraction can fail, retrieval can miss the useful passage, and the model can still improvise.

What I would build is narrower than “chat with all my documents”: a private retrieval pipeline whose evidence you can inspect.

This guide assumes Open WebUI already connects to Ollama. If it does not, build the isolated base first with Open WebUI on Windows and Ollama.

What Local RAG Actually Does

Open WebUI’s local document path has six practical stages:

file -> text extraction -> chunks -> embeddings -> retrieval -> answer

When you ask a question, the system embeds the query, searches the vector index for similar chunks, and injects selected text into the model prompt. The model then writes an answer from that context. Open WebUI’s RAG documentation describes this retrieval path and exposes controls for the splitter, chunk size, overlap, embedding model, reranking, and citations.

Each stage can be local:

• the original files stay on your server;
• extraction runs in Open WebUI or a local extraction service;
• the embedding model runs through Ollama or another local engine;
• vectors stay in the local database;
• the answer model runs through Ollama.

Verify the configuration rather than assuming “self-hosted UI” means every provider is local. A remote embedding endpoint still receives document text. A remote chat model still receives retrieved passages. The browser showing 192.168.x.x does not overrule the backend settings.

Start With a Small, Testable Collection

Importing 30,000 mixed files is a terrible first test. Create one Knowledge collection with:

• one text-based PDF;
• one scanned PDF;
• one short Markdown or text file;
• five questions whose answers you already know;
• two questions the documents cannot answer.

Use documents with stable page numbers and a few unique phrases. This makes extraction and citation errors obvious.

In Open WebUI, create a Knowledge base under Workspace, upload the files, and attach that collection to a chat or model. The official Knowledge guide distinguishes Focused Retrieval, which searches selected chunks, from Full Context, which injects the entire document. Full Context can be better for a small document that fits comfortably inside the model’s context. RAG is the practical choice for larger collections.

Ask the known questions first. Open each citation and compare the retrieved passage with the original page. Then ask the two unanswerable questions. A trustworthy setup should say the evidence is missing, not discover a previously unknown talent for contract law.

PDF Extraction Comes Before Embeddings

A vector index cannot recover text that was never extracted.

Text-based PDFs usually work with a basic parser. Scans and photos need OCR. Tables, multi-column layouts, headers, footers, and diagrams can scramble reading order even when every word is technically present.

Open WebUI supports several extraction engines and documents text PDFs, scanned PDFs, images, and handwritten documents in its document extraction guide. The correct engine depends on the files:

Document type	First approach	What to verify
Clean digital PDF	Built-in text extraction	headings, page order, tables
Scanned PDF	Local OCR engine	missing lines, columns, page numbers
Manual with diagrams	Structured extractor or OCR	captions and labels remain near the right section
Spreadsheet	Native tabular extraction	sheet names, headers, formulas versus displayed values

Before tuning embeddings, inspect extracted text from representative pages. If “battery cutoff voltage” became three disconnected columns, changing the chunk overlap will not repair it.

OCR also increases compute time and creates another copy of sensitive content. Keep extraction caches, temporary files, and backups inside the same storage policy as the originals.

Choose One Local Embedding Model and Keep It Stable

An embedding model converts text into vectors used for similarity search. It is not the chat model, and a larger chat model does not compensate for a poor or mismatched embedding index.

Configure a local embedding engine under Open WebUI’s document settings. Ollama can serve dedicated embedding models, so the document text does not need to leave the machine. Test multilingual retrieval if the collection mixes English, Portuguese, or technical identifiers.

The critical operational rule is simple: changing the embedding model requires reindexing. Vectors produced by different models occupy incompatible spaces. Open WebUI documents that existing Knowledge collections must be reindexed after an embedding model change; files uploaded only to individual chats need to be uploaded again.

Record these settings with the backup:

embedding engine and model
text splitter
chunk size
chunk overlap
reranker, if enabled
Open WebUI version

Otherwise a restore may produce a database that exists but retrieves nonsense.

Chunk for Retrieval, Not for a Round Number

Small chunks are precise but can lose the sentence before the answer. Large chunks preserve context but add noise and consume more of the prompt. Overlap protects boundaries while duplicating text and vectors.

A reasonable first experiment for ordinary prose is:

• use structure-aware splitting where headings are reliable;
• choose chunks large enough to contain one complete idea;
• add modest overlap for paragraphs that cross boundaries;
• retrieve a small set of passages and inspect them;
• change one variable at a time.

A universal “best chunk size” is not engineering. A legal contract, a service manual, and a recipe collection have different structure. Open WebUI supports character or token splitters and Markdown header splitting. Its documentation also explains a minimum chunk target that merges tiny fragments created by headings.

Build a small evaluation table:

Question	Expected source	Retrieved source	Answer supported?
What is the inverter shutdown voltage?	Manual, page 42
How long is the warranty?	Terms, section 8
Does the policy cover flood damage?	No source

Run it after changing extraction, embeddings, chunking, or the answer model. RAG quality should be measured against known evidence, not against how polished the prose sounds.

Make Unsupported Answers Uncomfortable

Use a system instruction that sets an evidence boundary:

Answer only from the retrieved sources.
Cite the source for each factual claim.
If the sources do not contain the answer, say that the evidence is insufficient.
Do not fill gaps from general knowledge.

This helps, but it is not enforcement. The model can ignore instructions, cite a weak passage, or attach a citation to a claim the passage does not support.

For decisions involving money, health, safety, or legal obligations:

1. open the cited passage;
2. verify the original document and version;
3. check whether the answer omitted a nearby exception;
4. make the decision from the source, not the summary.

Citations are a debugging interface. They are not a truth certificate.

Keep the Entire Data Path Local

Audit more than the model selector:

• embedding provider;
• document extraction provider;
• speech or image services;
• web search integration;
• external vector database;
• telemetry and update checks;
• backup destination.

If strict offline operation matters, block outbound access for the AI host after images and models are installed, then test ingestion and chat. Keep a controlled maintenance window for updates. This reveals hidden dependencies quickly.

The vector database deserves the same protection as the PDFs. Embeddings are not harmless random numbers; they encode semantic information and remain linked to extracted chunks and metadata. Encrypt the host drive, restrict Open WebUI accounts, isolate backups, and avoid giving every user access to every Knowledge collection.

For remote access, use the same VPN-first design as the base Open WebUI deployment. Publishing a private document assistant directly to the internet is a creative way to turn local RAG into public document retrieval.

Storage, Backup, and Recovery

Back up:

• original documents;
• Open WebUI application data;
• vector database data;
• extraction configuration;
• embedding model name and version;
• a copy of the evaluation questions.

An SSD matters more as the collection and index grow. Keep a second copy on an external drive, NAS, or encrypted off-site target. Do not assume regenerating the index is trivial until you have timed a complete reindex of the real collection.

After restoring, run the evaluation table again. A green container status only proves the process started. It says nothing about page extraction, vector compatibility, or whether the right passages come back.

The result is closer to a disciplined search-and-summarize tool than “your own ChatGPT for every PDF.” That is a feature. You can inspect the evidence, recognize failure, and verify that no document took a quiet trip to somebody else’s API.