Adaptive RAG: How to Route Queries Between Simple Retrieval and Agentic Pipelines

The problem with a single retrieval pipeline

Most teams build retrieval-augmented generation (RAG) the same way: embed the corpus, retrieve top-k chunks by cosine similarity, stuff them into a prompt, generate. It works well in a demo because demo queries are simple - "what's our refund policy," "summarize this document." Production traffic is not that homogeneous.

A support bot fields both "what's the return window on this SKU" (single-hop, answerable from one chunk) and "did the issue that caused the March 12th outage also affect the API rate limiter, and if so, what was the fix" (multi-hop, requires connecting two or three retrieved documents and possibly a follow-up query to fill gaps). If you run both through the same naive top-k retrieval, you get one of two failure modes:

• Under-provisioned for the hard query: single-pass retrieval misses the second or third hop, the model hallucinates a connection between the outage and the rate limiter that doesn't hold up, and the answer is confidently wrong.
• Over-provisioned for the easy query: you route everything through an agentic loop - plan, retrieve, evaluate, retrieve again, synthesize - and a one-line factual lookup that should cost one embedding call and one generation call now costs 3-5 LLM round trips and multiple retrieval hits. Agentic RAG adds meaningfully higher latency and cost over naive retrieval on queries that didn't need it.

Adaptive RAG addresses this directly: classify the query first, then route it to the retrieval strategy that matches its actual complexity.

Architecture: what adaptive routing actually looks like

The core idea is a lightweight classification stage inserted before retrieval:

User query
    |
    v
Query Classifier (small/fast model or rules)
    |
    +-- No retrieval needed -> direct generation (greeting, chit-chat, math)
    |
    +-- Simple factual lookup -> single-pass vector search -> generate
    |
    +-- Comparative / multi-hop -> decompose into sub-queries ->
    |       parallel retrieval per sub-query -> merge -> generate
    |
    +-- Open-ended / exploratory -> agentic loop:
            plan -> retrieve -> self-critique/reflect -> retrieve again
            (bounded iterations) -> synthesize

The classifier

This is the component that makes or breaks the pattern, and it's worth being deliberate about it rather than bolting on a single "is this complex?" LLM call.

Option 1 - small fine-tuned classifier. Train a lightweight model (a distilled BERT-class model or a small open-weight model) on labeled query/complexity pairs. Cheap and fast (single-digit milliseconds), but requires you to build and maintain a labeled dataset, and it needs periodic retraining as your query distribution shifts.

Option 2 - LLM-based routing with a small model. Use a fast, cheap model (something in the Haiku/Flash/Mini tier) with a structured-output prompt that returns a complexity label and a confidence score. Higher latency than a fine-tuned classifier (typically 100-300ms) but zero training data requirement and easier to iterate on - you change the prompt, not retrain a model.

Option 3 - heuristic pre-filter, model fallback. Route obviously simple queries (short, single entity, matches a FAQ-like pattern) via cheap heuristics, and only invoke the LLM classifier for anything ambiguous. This is the most cost-efficient in practice because a large fraction of production traffic really is simple, and heuristics catch it for free.

Most production systems converge on a hybrid: heuristics catch the obvious cases, a small fast model handles the rest, and only genuinely ambiguous or low-confidence classifications fall through to a more expensive path (or get flagged for a stricter, slower retrieval strategy as a safe default).

# Simplified router sketch
def classify_query(query: str) -> RouteDecision:
    if is_greeting_or_chitchat(query):
        return RouteDecision(route="direct", confidence=0.95)

    if matches_faq_pattern(query) and entity_count(query) <= 1:
        return RouteDecision(route="simple_retrieval", confidence=0.85)

    # Fall through to fast LLM classifier for anything not caught above
    result = fast_classifier_llm.classify(
        query,
        labels=["simple", "multi_hop", "exploratory"],
        return_confidence=True,
    )

    if result.confidence < 0.6:
        # Low confidence - default to the safer, more thorough path
        return RouteDecision(route="multi_hop", confidence=result.confidence)

    return RouteDecision(route=result.label, confidence=result.confidence)

Query decomposition for multi-hop routes

For queries classified as multi-hop, the retrieval step itself needs to change, not just get slower. Decompose the query into sub-questions, retrieve for each independently (in parallel, not sequentially, to control latency), then merge and deduplicate before generation:

Don't default every "complex-looking" query to the sequential agentic loop. A large share of what looks like multi-hop is actually parallel-decomposable, and that distinction alone accounts for much of the latency difference between a well-tuned adaptive system and a naive "always agentic" one.

Failure modes and how to design against them

Misclassification cascades. If the classifier under-routes a genuinely complex query to simple retrieval, the failure is silent - you get a plausible-sounding, wrong answer with no signal that anything went wrong. Mitigate this by tracking a proxy signal downstream: if the generation step's self-reported confidence is low, or if the retrieved chunks have low similarity scores despite being "top-k," escalate to the next tier rather than returning the answer as-is. Treat routing as provisional, not final.

Classifier drift. Query patterns change as your product changes - a new feature launches, support tickets shift topic, users start asking longer or shorter questions. A classifier tuned on last quarter's traffic distribution silently degrades. Re-evaluate classifier accuracy against a held-out sample of production queries on a fixed cadence (monthly is a reasonable default for most teams), not just at initial launch.

Latency variance becomes a UX problem. Once different queries take meaningfully different amounts of time, users notice the inconsistency more than they'd notice a uniformly slower system. If your product surface expects a chat-like response, consider streaming partial retrieval status ("searching related documents...") for anything routed to the slower paths so the variance reads as thoroughness rather than a stall.

The router itself becomes a bottleneck. If the LLM-based classifier sits in the critical path for every single query, its own latency and cost start to matter. This is precisely why the heuristic-first tier exists - most traffic should never touch the LLM classifier at all.

Over-fitting the router to your eval set. It's easy to tune the classifier against a golden set until it looks great, then watch it perform worse on live traffic because the golden set doesn't represent the actual query distribution. Build the golden set from sampled production logs, not from queries your team imagines users will ask.

A decision framework

Use adaptive routing when:

• Query complexity in your traffic is genuinely bimodal or multimodal - you have both simple lookups and multi-hop questions in meaningful volume.
• Cost or latency budgets are tight enough that uniformly agentic retrieval isn't viable at your traffic volume.
• You can instrument the pipeline well enough to catch misclassification (confidence scoring, escalation paths, monitoring).

Skip it and keep a single pipeline when:

• Your query distribution is genuinely uniform (e.g., a narrow internal tool with one query shape).
• Your traffic volume is low enough that the cost difference between "always agentic" and "adaptive" doesn't matter in absolute dollars.
• You don't yet have the observability to know what's failing - adding routing complexity on top of an unmonitored pipeline just adds another place for silent failures to hide.

A pragmatic build order: ship naive single-pass RAG first, instrument it (log query, retrieved chunks, generation output, and a human or LLM-judge quality score), let two to four weeks of production traffic accumulate, then look at the failure distribution. If failures cluster on a recognizable subset of queries (comparative, multi-part, or long-tail entity questions), that's your evidence for where to route to a second tier - and you'll have a real labeled dataset to build the classifier from instead of guessing.

Evaluation: how to know it's working

Adaptive routing needs two evaluation layers, not one:

1. Routing accuracy: does the classifier send queries to the tier that actually matches their complexity? Build this from a sampled, human- or LLM-judge-labeled set of production queries, and track it as its own metric separate from end-to-end answer quality.
2. End-to-end answer quality per route: faithfulness and relevance scores (via LLM-as-judge or human review) computed separately for each route. A common trap is looking only at the aggregate quality score across all traffic - that number can look fine even while the multi-hop route is quietly underperforming, because it's a small fraction of total volume.

Track cost and latency per route as first-class metrics too. The entire point of the pattern is the cost/latency curve, so if you're not measuring p50/p95 latency and per-query cost broken out by route, you can't tell whether the added complexity of a router is actually paying for itself versus just adding an extra hop with no measurable benefit.

Sources

• Production-Ready RAG Architecture Patterns for 2026
• Building Production RAG: Architecture, Chunking, Evaluation & Monitoring
• Agentic RAG in 2026: Architecture Patterns, Frameworks & When to Use It
• 10 RAG Architectures in 2026: Enterprise Use Cases & Strategy

Syslabs' engineering team builds retrieval-augmented generation and agentic pipelines like this as part of production LLM systems for clients - including the query routing and evaluation instrumentation described above.