RAG retrieves knowledge; it does not supply reasoning.

Retrieval-augmented generation answers from content pulled out of an organization's own documents. Where no document holds the answer, where documents contradict each other, or where the question needs inference across several steps, the approach does not carry the load alone.

Short answer: RAG is a retrieval layer. Its limit is whether the answer is written down somewhere.

Let's classify your RAG use case See the Enterprise RAG service

Where RAG genuinely works

RAG is strong where the answer is written in a document and the source changes often: procedures, product documentation, contract clauses, policy texts. Because the model is not retrained, the answer moves when the source moves, and the citation can be checked.

Accuracy is high when one document holds the answer
A changed source needs no retraining
The citation makes the answer auditable
Permission filters are inherited from the source system

The four cases where it falls short

Failed answers rarely share one cause; they usually split into four. Without that split, teams repair the wrong layer — swapping the model does nothing for a retrieval problem.

The knowledge was never written down: no retrieval can find it
Documents contradict: someone must decide which one governs
The question needs multi-step inference, not a single lookup
The answer needs computation: a tool, not a model, produces the number

The next step: training or tools?

Once the missing piece is named, the remedy follows. Missing behavior or format points to supervised fine-tuning, missing domain language to continued pre-training, and missing computation to tools the model can call. These are not alternatives to one another; each answers a different gap.

Fine-tuning for format and task behavior
Continued pre-training for depth in domain language
Tool calling for computation and queries
A planning workflow for multi-step tasks

Measurement makes the decision

Which path to take is settled by an evaluation set drawn from real questions, not by discussion. The same set runs before and after the change, and nothing ships while a regression stands.

The set is built from real user questions
Retrieval accuracy and answer correctness score separately
Declining to answer counts as a form of success
Every change is measured against its own baseline

How we work

Collect the failed answers rather than guessing at them
Assign each failure to one of the four causes
Separate missing knowledge from missing behavior and computation
Match each class to its own remedy
Confirm the decision with the evaluation set

How success is measured

The share of uncited answers is falling
Unanswered questions are deliberate and traceable
Failures can be classified by cause
Every change is measured against the same baseline

Frequently asked questions

What should RAG do when no document exists?

It should not answer. Silence beats invention when there is no source, and that behavior belongs in the system design. The real fix is writing the knowledge down.

Does fine-tuning replace RAG?

No. Fine-tuning changes behavior and format; it does not keep knowledge current. Frequently changing organizational knowledge needs a retrieval layer, and most organizations end up using both.

What do multi-step questions require?

A workflow that plans the steps and can call tools. A single retrieval pass cannot serve questions that need intermediate results.

Share a sample of your failed answers and we will classify which layer the gap sits in.

Let's classify your RAG use case