The problem
Retail investors ask two kinds of question about a mutual fund. One kind has an answer printed on the scheme page — expense ratio, exit load, minimum SIP. The other doesn't: should I invest, which fund is better, what returns should I expect.
A general chatbot answers both, confidently. The second kind is where it becomes a liability — in India, investment advice is regulated activity.
Facts Desk answers the first kind and refuses the second. Ask for an expense ratio and you get one sentence, one citation from a five-URL allow-list, and the date that source was last fetched. Ask whether to invest and you get a refusal pointing to SEBI.
The decision I'd defend
Safety lives in what gets stored, not in what gets said.
The obvious approach is to filter the output — let the model see everything, then catch anything advisory on the way out. That works until someone finds a phrasing the filter didn't anticipate.
Instead, refusable content is excluded at ingestion. Holdings tables, "compare similar funds" sections, and every performance section are never indexed. The system cannot leak a return figure under adversarial prompting, because no return figure exists anywhere in its index. A filter can be talked around. Absent data cannot.
Behind that sit a deterministic validator — sentence count, citation count, allow-list membership, and every number verified verbatim against the cited source — and a groundedness check against the specific chunk that was cited. The model is asked to comply. The validator ensures it.
The number that taught me the most
Factual accuracy on the first evaluation run read 18.8%.
My instinct was to rewrite the prompt. Instead I looked at the failures — and they weren't wrong answers. They were non-answers. Groq's free tier was rate-limiting an eval firing 32 queries back to back, and those API errors were falling into the safe-fallback path, where they became indistinguishable from genuine coverage gaps.
The measurement was broken, not the system.
Separating infrastructure failures from content failures took it to 65.6%. Two real bugs then surfaced: an alias resolver rejecting two of my own in-corpus schemes as out-of-scope, and under-specified answers on compound attributes. Fixing both reached 96.9%.
At no point did I lower a threshold to make a number move. That rule is why every jump came from finding a defect rather than redefining success.
Results and limits
31/32 factual accuracy on a human-verified golden set. 100% citation validity. 42/42 refusal recall, including compound queries that bundle a factual ask with an advisory one — the most common leak in facts-only systems.
Retrieval reports recall@5 of 1.000, and I say in the repo why that's meaningless: with ten chunks, top-5 returns half the corpus. The honest figure is recall@1 at 0.844.
The confidence threshold isn't tuned. The scheduler has never been observed at a live tick. One golden item still fails groundedness. All of it is documented per phase — a system with stated gaps is more trustworthy than one claiming none.
Stack
BGE embeddings · ChromaDB · BM25 hybrid retrieval · cross-encoder reranking · Groq inference at temperature 0 · FastAPI with SSE streaming · append-only audit log