Why production knowledge often lives in systems, not PDFs

Asking a document index "how many open tickets do we have right now?" is like asking a library for the current time. Wrong building, and it will helpfully give you an answer anyway: probably from a status report written in April.

Live facts do not live in PDFs. Inventory, order status, account balances, ticket counts, entitlements: these live in databases and business systems, and they change faster than any index can track.

Two retrieval modes, one chat box

Document retrieval answers what does the policy say. Structured retrieval answers what is currently true.

Production assistants need both, and the moment you add the second one, three problems appear that document-only RAG never had.

Routing. Something must decide, per question, whether the answer lives in prose or in a table. Get it wrong and the system quotes a stale PDF about a number the database knows exactly. This routing decision deserves to be explicit and logged: when answers are wrong, "it asked the wrong system" and "it asked the right system badly" need different fixes.

Query safety. Model-generated SQL running against production with write access is an incident generator with a natural-language interface. The floor is: read-only credentials, parameterised templates or narrow typed tools rather than free-form SQL, allowlisted views instead of raw table access, row limits, and query timeouts. Prefer exposing five specific tools (get_order_status, get_open_ticket_count) over one general run_query tool. Narrow tools are easier to validate, easier to log, and dramatically harder to misuse.

Freshness versus cost. Live calls are current but slower, rate-limited, and occasionally down. Decide per question class whether staleness or latency is the greater sin, and design the fallback for when the system of record is unavailable. "The database timed out" should not silently become "answered from a cached document."

Evidence has provenance here too

An API result is evidence like any other, and it needs the same treatment: what was queried, against which system, at what time, returning what.

Without that, a wrong answer is unfalsifiable after the fact. You cannot tell whether the query was wrong, the data was wrong, or the data was right at the time and has since changed. All three happen; only logging distinguishes them.

The demo question is "what is our refund policy?" The production question is "where is my refund?"

They arrive in the same chat box, phrased almost identically. They are answered by completely different infrastructure, and a system that only built one of them will answer the other confidently and wrongly.

Which of your answers is being served from a stale document when a live system knows the real number?