An assistant grounded in your own documents is usually sold as a model problem. It is almost always a retrieval problem. The model is the last step in the pipeline and the most visible one, so it takes the blame for an answer that was already decided several steps earlier, at the point where the system chose which paragraphs to look at.
This matters because the two failures look identical from the outside. A customer reads a confidently worded answer about a policy that was withdrawn in March. The model did nothing wrong: it was handed the March document and summarised it accurately. The mistake was upstream, and no amount of prompt tuning will fix it.
What retrieval actually does
A grounded system (usually called retrieval-augmented generation, or RAG) runs three steps in order:
- Turn the question into a search. The user's words are converted into one or more queries, often an embedding plus a keyword search, sometimes a structured filter as well.
- Fetch and rank candidate passages. The index returns the passages most similar to the query, re-ranks them, and keeps the top handful that fit in the model's context window.
- Answer from what was fetched. The model is instructed to use only the retrieved passages, cite them where useful, and say it does not know when they do not contain the answer.
Step three is the part everyone demos. Steps one and two are where accuracy is decided. If the right paragraph never enters the context window, there is no prompt that recovers it.
The ceiling on a grounded system's accuracy is the proportion of questions for which the correct source material reaches the model. Everything the model does afterwards can only lower that number, never raise it.
Chunking is a content decision, not a parameter
The most common default is to split every document into fixed-size chunks of some number of tokens with a small overlap. It is easy to implement and it quietly destroys structure.
A fee schedule split at 800 tokens can separate the fee from the condition that qualifies it. A policy split mid-clause can retrieve "this does not apply" without the sentence that says what "this" is. A table split across two chunks returns headers without rows, or rows without headers.
Chunk on the document's own boundaries wherever the format gives you any:
- Split on headings, clauses, list items or table rows rather than on a token count.
- Keep the heading path on the chunk, so a passage carries
Refunds > International orders > Exceptionswith it. - Keep whole tables together even when they are long, or convert each row into a self-contained sentence before indexing.
- Overlap deliberately. Enough to carry a pronoun across a boundary, not so much that near-duplicates crowd out the top results.
The test is simple: read a chunk on its own, with no surrounding document. If you cannot tell what it is about or what it applies to, the model cannot either.
Give every chunk its context back
A retrieved passage should be able to explain itself. Prepend a short header to each chunk at index time, the document title, the section path, the effective date, the jurisdiction or product it applies to. It costs a few tokens and it lets the model notice that the passage it was handed is about the wrong product.
Metadata is what makes retrieval correct, not just relevant
Similarity search answers "what text looks like this question." It does not answer "what is currently true for this customer." Those are different questions, and conflating them is the single most common cause of confidently wrong answers.
Attach structured fields to every chunk at index time and filter on them before ranking:
| Field | What it prevents |
|---|---|
effective_from / effective_to |
Answering from a superseded policy that still scores well on similarity |
status (draft, approved, archived) |
Quoting a document nobody approved |
audience or region |
Giving a customer in one market the terms that apply in another |
product / plan |
Mixing tiers that use nearly identical wording |
source_system |
Being unable to say where an answer came from when it is challenged |
Filtering first and ranking second is usually a larger accuracy improvement than any change to the embedding model. It also makes failures explainable: you can show which filters were applied and why a document was excluded.
Hybrid search, because your users type identifiers
Pure vector search is strong on paraphrase and weak on exact strings. Real questions contain exact strings constantly: an order number, an error code, a SKU, a clause reference, a surname. Embeddings blur those into something approximately similar, which is the opposite of what the user wanted.
Run keyword search and vector search together, then merge the result lists. Keyword matching catches ERR_CONN_4420 and clause 7.3; vector matching catches "my connection keeps dropping" and "what happens if I cancel early." Combining the two is a small amount of engineering and it removes an entire class of misses.
Then re-rank. A cross-encoder re-ranker scores each candidate against the full question rather than against a compressed vector, and it is good at pushing the one genuinely relevant passage above four that merely share vocabulary. Retrieve generously, twenty or thirty candidates, and let the re-ranker decide which five the model sees.
The source corpus is part of the system
Retrieval cannot resolve a contradiction that exists in your documents. If two approved pages state different refund windows, a grounded assistant will cite one of them, and which one it cites is close to arbitrary.
This is why a knowledge assistant project usually turns into a content project somewhere in the first month. The work that actually moves accuracy tends to be unglamorous:
- Find and retire duplicate documents, including the three slightly different copies in shared drives.
- Decide which system is authoritative for each class of fact, and index only that one.
- Remove internal drafts, meeting notes and superseded versions from the index entirely.
- Give every document an owner and a review date.
- Write down the facts that exist only in people's heads, because those are the questions users ask most.
Refusal is a feature you have to build
The hardest behaviour to get right is the one that produces no answer. A grounded system should distinguish three outcomes: it found the answer, it found something adjacent but not sufficient, and it found nothing relevant at all.
Most systems collapse the middle case into the first. The retrieved passages are "close enough," the model is helpful by default, and the result is a fluent answer assembled from material that does not actually support it.
Build the boundary explicitly:
- Set a relevance floor from your re-ranker score, below which the system does not attempt an answer.
- Instruct the model to answer only from the supplied passages and to name what is missing when it cannot.
- Require a citation for any claim that carries a number, a date, a price or an obligation.
- Route an unsupported question to a human with the question and the retrieved context attached, so the person is not starting from nothing.
Saying "I don't have an approved source for that, I've passed it to the team" is a correct answer. Treat it as a success in your metrics, not a failure, or you will optimise the refusal away.
Measuring retrieval separately from generation
Evaluate the two halves apart, because they fail for different reasons and have different fixes.
For retrieval, assemble a set of real questions (pulled from support tickets, search logs and the inbox, not invented), and label each with the document and section that should answer it. Then measure how often the correct source appears anywhere in the retrieved context. That single number is your accuracy ceiling.
For generation, take only the questions where retrieval succeeded and ask whether the answer is faithful to the passages supplied, whether every factual claim is supported, and whether it refused correctly when the passages were insufficient.
Keep a third set of questions the system should never answer (pricing exceptions, legal interpretation, anything requiring judgment), and check that each one escalates rather than improvises.
Run all three on every change: a new embedding model, a different chunking strategy, a re-ranker swap, a prompt edit, a model upgrade. Without that, you are changing a system whose behaviour you cannot compare to last week's.
Where to start
If you have a grounded assistant that is wrong more often than you can defend, work in this order:
- Sample twenty wrong answers and classify each one: wrong source retrieved, right source retrieved but ignored, or no source existed.
- If most are the first category, fix chunking and metadata filtering before touching the model.
- If most are the second, tighten the answer instructions and add citation requirements.
- If most are the third, you have a content problem, and the assistant is telling you which documents to write.
That classification takes an afternoon and it almost always redirects the work. The temptation is to upgrade the model, because it is one line of configuration. The accuracy is in the index.
If you are building or repairing a grounded assistant, see how we scope AI systems or book a strategy call to walk through your corpus before choosing an architecture.