By Vamsi Krishna
Engineering · · Discuss · updated
60 Days of Production AI Systems · chapter 4 of 10Evidence-Driven RAG Patterns Day 23 · part 5 of 6

New here? Start this series at Day 19: Why reranking is where retrieval becomes useful

  1. Day 19: Why reranking is where retrieval becomes useful
  2. Day 20: Why every important AI claim needs provenance
  3. Day 21: Why RAG needs both broad context and precise evidence
  4. Day 22: Why one user question may need many searches
  5. Day 23: Why complex AI questions need decomposition
  6. Day 24: Why some answers live in relationships, not documents

"Compare our Q3 churn against our top two competitors and suggest what to fix."

That is five questions in a trench coat, and no single retrieval pass answers it well.

What a bundled question does to retrieval

Embed that request and you get a point in vector space that sits near everything and matches nothing well, an average of churn, competitors, and remediation that corresponds to no document you own.

So retrieval returns a blend: mostly-relevant material for the loudest part of the question, thin coverage of the rest. The model then writes a fluent paragraph across all five topics, with solid grounding on one and improvisation on four.

The output reads as complete. That is the whole problem. Nothing in the answer flags that "our competitors' numbers" came from nowhere.

Splitting it makes the work inspectable

Decomposition identifies the sub-questions first, retrieves for each independently, and merges only once each part has its own support.

The retrieval gain is obvious: each search finally has one clear target. The operational gain matters more: every sub-answer becomes a checkpoint. When the final output is wrong, you can point at the step that failed rather than staring at one opaque paragraph wondering which third of it to distrust.

One broad request, turned into inspectable work

User asks: "Can we safely automate this support refund workflow?"

  1. What does the refund policy actually permit?
  2. What is this customer's and order's current state?
  3. Which actions are within our authority to take automatically?
  4. Draft the response.
  5. Record evidence, decision, and owner.

Each step has its own evidence, its own success criteria, and its own failure mode. Step 2 failing looks nothing like step 1 failing, and in a bundled answer, both look identical.

The two ways it goes wrong

Decomposing everything. Simple questions do not deserve a five-step plan; they deserve an answer. A system that fans out "what is your phone number" into sub-questions has traded latency and cost for nothing. Route by complexity, and make that routing decision explicit rather than emergent.

Unbounded depth. Sub-questions that spawn their own sub-questions need a depth limit and a cost ceiling. Without them, one genuinely hard query can quietly consume an afternoon of compute, and the trace will show it was reasonable at every individual step.

The harder skill is judging when a question is actually a project.

Encode that judgement in the routing layer. Applying it uniformly is just a more expensive way to answer easy questions.

Which request in your product is really several questions hiding inside one sentence?

Checklist · · 12 checks

RAG production-readiness checklist

Twelve checks to pass before a retrieval system answers a real user. Tick them locally; progress stays in your browser.