Is RAG still worth it with long-context models? (2026)

If you search whether RAG is still worth it now that models read a million tokens, you mostly find two kinds of answer. One says "RAG is dead, just paste everything in." The other is a 2024 comparison built on 128K-token windows and long-context surcharges that no longer apply. Neither helps you decide for your own corpus in late 2026.

This post covers what changed, what the research says about long prompts, a six-question test for choosing between long context and retrieval, and what a modern RAG pipeline looks like when you do need one.

What changed in 2026

Context windows are now large and, on some APIs, no more expensive per token at the top end. Anthropic lists Claude Opus 5.5 and Sonnet 5.5 with a 1M-token context window (models overview), and its pricing page says these models bill a 900K-token request "at the same per-token rate as a 9k-token request" (Claude pricing). Google lists an input limit of 1,048,576 tokens for Gemini 3.1 Pro Preview (model page), but still charges twice as much for prompts over 200K tokens ($4 instead of$2 per million input tokens) (Gemini pricing).

So "it fits" is true for far more corpora than it used to be. Fitting is not the same as working well, or being cheap.

What the research says about long prompts

Three results are worth knowing:

• Position matters. Liu et al. found accuracy is highest when the relevant passage sits at the start or end of the prompt and "significantly degrades" when it sits in the middle (Lost in the Middle, 2023).
• Claimed length is not usable length. NVIDIA's RULER benchmark found that of models claiming 32K or more tokens, "only half" kept satisfactory performance at 32K (Hsieh et al., 2024).
• The decline is still visible on recent models. Chroma tested 18 models, including GPT-4.1, Claude 4 and Gemini 2.5, and found performance "grows increasingly unreliable as input length grows," even on simple tasks (Context Rot, July 2025).

There is a fair counterpoint. Li et al. found that long context beat RAG on average answer quality "when resourced sufficiently," while RAG kept a large cost advantage (Li et al., 2024). Their fix, Self-Route, tries cheap RAG first, lets the model say when the retrieved context is not enough, and sends only those queries to full long context. They reported cost cuts of 65% for Gemini-1.5-Pro and 39% for GPT-4o at comparable quality. They come from 2024 models, so treat them as the shape of the trade-off.

A worked cost example

Take an 800K-token document set on Claude Sonnet 5.5, at $2 per million input tokens (Claude pricing):

At 10,000 queries a day that is roughly $16,000,$1,600 or $160 in input tokens. Prompt caching narrows the gap a lot, but cache writes cost more than plain input (1.25x for the 5-minute cache, 2x for the 1-hour cache), and a cache only helps when the same prefix repeats. The table also leaves out latency: the model still has to process a long prompt, while a RAG prompt stays small.

Six questions: long context or retrieval?

This checklist is our own heuristic, not a vendor rule. Answer the questions in order.

1. Size. Does the corpus fit with room to spare, and will it still fit next year? Anthropic's own guidance is that a knowledge base under about 200,000 tokens (around 500 pages) can go straight into the prompt (Contextual Retrieval). Above that, retrieval earns its place quickly.
2. Cost. Multiply corpus tokens by queries per day. If the number is uncomfortable even with caching, retrieve.
3. Latency. If users need a fast first token, a small prompt helps.
4. Freshness. If documents change hourly, re-indexing changed chunks is cheaper than rebuilding and re-caching one giant prompt.
5. Citations. If answers must point to a source passage, retrieval gives you chunk IDs to cite. With a single huge prompt you have to ask the model to quote, then check that it did.
6. Access control. If different users may see different documents, you need to filter before the model sees anything. OWASP recommends "permission-aware vector and embedding stores" for this reason (OWASP LLM08:2025). A shared long prompt cannot enforce per-user permissions.

If long context passes all six, skip RAG. Most production systems fail at least one.

What modern RAG looks like

The 2023 tutorial version (fixed-size chunks, one embedding model, top-5 cosine similarity) is not the bar anymore. A current pipeline usually has four parts.

Hybrid search. Run dense vector search and keyword search (BM25) together and merge the results. On the BEIR benchmark, BM25 proved "a robust baseline" across domains (Thakur et al., 2021). It catches exact names, codes and error strings that embeddings blur. Reciprocal rank fusion is a simple, training-free way to merge the two ranked lists (Cormack et al., SIGIR 2009).

Contextual chunking. Split on document structure, not fixed windows, and add a short summary of where each chunk sits before you embed it. In Anthropic's tests, contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%, from 5.7% to 2.9% (Contextual Retrieval).

Reranking. Retrieve a wide candidate set, then re-score it with a cross-encoder or late-interaction model and keep the best few. Adding reranking took Anthropic's failure rate to 1.9%, a 67% reduction (Contextual Retrieval). BEIR found reranking and late-interaction models gave the best zero-shot results "at high computational costs" (Thakur et al., 2021), which is why they run on a short list, not the whole index.

Evals. Measure retrieval and generation separately. For retrieval, track recall@k on a labeled set of questions. For generation, frameworks such as RAGAS score faithfulness to the retrieved context without needing human-written reference answers (Es et al., 2023). Without both, you cannot tell whether a bad answer came from a missed chunk or from the model ignoring a good one.

Long context still has a place in this design: as the escalation path for the hard queries, as in Self-Route, rather than the default for every query.

Learn it hands-on

RAG: Retrieval-Augmented Generation, end to end is a 16-lesson course for engineers who know Python and have called an LLM API, with no prior RAG background needed. It covers RAG foundations (the full pipeline, hybrid search, and failure modes), embeddings and indexing (choosing models with MTEB, HNSW and IVFFlat tuning, fine-tuning retrievers), retrieval quality (contextual chunking, reranking, query transformation such as HyDE), advanced and agentic RAG (Self-RAG, Corrective RAG, GraphRAG, a full long-context vs. RAG lesson on Self-Route, and a production case study), and multimodal retrieval over images, tables and PDFs.

Foundations lessons include runnable Python you can edit in the browser. Lessons cite the original papers, and you can review with LearnOS flashcards or ask the lesson chat, which answers from the course's own sections.