RAG that refuses to guess

The support agent handles orders, delivery windows, returns and payments for an Arabic fashion platform. It runs on Qwen3-VL-8B, quantized to 4 bits, served on a single T4. Ask that model about a return policy it cannot find in its context and it will produce one. Fluently. In the customer's own dialect, with the confident cadence of a policy that exists.

For a bot that quotes refund terms to people spending real money, that is not a quality problem. It is a liability. So the agent is built on one rule that has nothing to do with prompt engineering: the model never gets to decide whether it failed.

The shape of the graph

The graph has five nodes that do real work. The agent node calls the LLM. The tools node executes tool calls. The validate node runs once the model stops calling tools. force_escalate synthesizes an abstention and hands off to a human. And human_review is a pure passthrough that returns an empty dict, existing only as the target of a native LangGraph interrupt.

Retrieval runs three legs

Every factual answer is grounded in a Qdrant collection with three independently queryable named vectors, populated from the same source documents.

support_docs
  dense      1024-d, cosine   Qwen3-Embedding-0.6B, served on the same vLLM host
  sparse_en  IDF-modified     fastembed Qdrant/bm25, English stemmer
  sparse_ar  IDF-modified     fastembed Qdrant/bm25, Arabic stemmer

A character-ratio detector routes the query before anything runs. A pure-English question skips the Arabic sparse leg; a pure-Arabic one skips the English leg; anything mixed runs all three. The dense leg never gets a language filter, which is deliberate: an Arabic question should still be able to retrieve an English-language policy document.

Surviving legs fetch their top 20 concurrently under a 2-second timeout each. A leg that times out is dropped and logged rather than failing the turn, then Reciprocal Rank Fusion merges whatever came back.

# Cormack et al. 2009, k = 60
fused[chunk] = sum(1 / (60 + rank_in_leg) for leg in surviving_legs)

The fused top 12 goes to a BGE cross-encoder running on a separate CPU host, which trims to the four chunks that actually enter the prompt. That reranker is fail-open: if it times out, the RRF ordering is used instead. A slow reranker degrades ranking quality. It does not take the agent down.

Three ways to fail safely

Three conditions route the turn away from the model entirely. Each one stamps a reason onto graph state, because routing edges cannot carry side-channel data.

• retrieval_unavailable: every leg failed, or fusion returned zero candidates. There is nothing to ground an answer in.
• below_score_floor: retrieval returned something, but the top reranked chunk scores under RERANK_MIN_SCORE. Retrieval worked and still found nothing worth answering from.
• answer_validation_failed: the model produced a reply, and a post-hoc check rejected its citations.

The second one is worth dwelling on. It is an answerability gate, and it catches the most dangerous case in retrieval-augmented generation: the query that returns plausible neighbours. Ask about a wholesale return policy when only the retail one is indexed, and cosine similarity will happily hand back the retail chunk. It is topically adjacent and factually wrong, and without a floor the model will paraphrase it into an answer.

The citation check

The system prompt requires every factual sentence to carry a [chunk_id] marker pointing into the assembled context. After generation, that requirement is checked mechanically rather than trusted.

def validate_answer(reply: str, context_ids: set[str]) -> str | None:
    for sentence in factual_sentences(reply):
        cited = CITATION_RE.findall(sentence)
        if not cited:
            return "answer_validation_failed"      # asserted without a source
        if any(cid not in context_ids for cid in cited):
            return "answer_validation_failed"      # cited a chunk we never sent
    return None

Where the three routes converge

All three reasons lead to force_escalate, which never lets the model see or paper over the failure. It writes a fixed abstention reply plus a signal ToolMessage, then routes unconditionally to human_review.

agent           --tool_calls present--------->  tools
agent           --no tool_calls-------------->  validate

tools           --RETRIEVAL_UNAVAILABLE------>  set_retrieval_unavailable
tools           --BELOW_SCORE_FLOOR---------->  set_below_score_floor
tools           --otherwise------------------>  agent

validate        --citations failed----------->  force_escalate
validate        --otherwise------------------>  END

force_escalate  ----------------------------->  human_review   (interrupt here)
human_review    ----------------------------->  agent          (after admin reply)

That interrupt is the real one from LangGraph, not a sleep-and-poll loop. When the graph pauses, the API layer pulls the summary and transcript off graph state, opens a Slack thread, and stores the channel and thread timestamp on the session. Later customer messages append to that same thread. When the admin replies, the graph resumes from exactly where it stopped.

Measuring the right thing

The dashboard splits escalations into two groups, and the split is the whole point. Genuine handoffs are retrieval_unavailable and explicit agent escalations: cases where no answer existed. Avoidable escalations are below_score_floor and answer_validation_failed: cases where the bot had grounding material and still could not produce a defensible answer.

Avoidable escalations are the number that drives the knowledge base backlog. Genuine ones mostly mean a human should have been involved anyway.

4

chunks survive into the prompt

2s

timeout per retrieval leg

0

unvalidated drafts shown to customers

A background task classifies each escalation using the full transcript rather than the escalating turn, because the escalating turn is usually just 'yes, connect me' and carries no topic signal at all. Classification failures fall back to a default label instead of dropping the queue row.

The honest tradeoff sits in one config value. Raise RERANK_MIN_SCORE and more turns reach a human; lower it and weaker answers reach customers. That dial is visible, tunable, and attached to a metric. When a customer complains about an answer, I can open the turn, see which route fired, and read the exact chunks the reranker chose. That traceability is the reason the decision lives in graph routing instead of inside a prompt.