Skip to content

Long Context vs RAG for Customer Support in 2026: When a Full Prompt Beats Retrieval

Placing a whole help center in every prompt is simpler but costs more per question. See the token math, cache effects, and when retrieval fits better.

Asad AliFounder & CEO
Updated
Reading time
12 min read

For most customer support, retrieval (RAG) is still the better default: it sends a few thousand tokens per question instead of the whole knowledge base, so it is far cheaper and usually faster. Long context wins when the knowledge base is small, volume is low, or a single long document has to be read as a whole. Context windows are now large enough that the choice is real: Anthropic lists a 1M-token window for Claude Sonnet 4.6, and OpenAI lists 1.05M tokens for its GPT-6 models.

This article works through the cost math with prices from the vendors' official pages, checked November 12, 2026. Chatsy publishes this article and sells an AI support agent; we explain how Chatsy handles this at the end.

Quick definitions

RAG (retrieval-augmented generation): Index your knowledge base into a search index (often a vector database such as pgvector, Pinecone or Weaviate, sometimes combined with keyword search). At question time, find the most relevant passages, put them in the prompt and generate the answer. The model sees a few thousand tokens of source text per request.

Long context: Skip retrieval. Put the entire knowledge base in the prompt every time. The model reads everything and works out which parts matter. The prompt can be hundreds of thousands of tokens.

Both produce an answer grounded in your content. The cost, latency and failure modes differ a lot.

Context windows and prices, checked November 12, 2026

ModelContext windowInput per 1M tokensCache hit per 1MOutput per 1M
Claude Sonnet 4.61M$3$0.30$15
Claude Haiku 4.5200k$1$0.10$5
Claude Haiku 5.51M$0.10 (prompts up to 100k tokens), $0.50 (over 100k)$0.01, $0.05 over 100k$0.50, $2.50 over 100k
GPT-5400k (272k input)$1.25$0.125$10

Sources: Anthropic pricing, Anthropic context windows, OpenAI pricing, OpenAI GPT-5 model page. Anthropic says Claude 4.6 and later models (except Haiku 5.5) bill the full 1M window at standard rates. OpenAI marks GPT-5 as deprecated, with API removal on December 11, 2026, so treat its row as a reference point.

For scale: a help center of 500 articles averaging 600 words is about 300,000 words, or roughly 400,000 tokens at about 1.33 tokens per English word. That fits in a 1M window but not in a 200k one.

The cost math

Take a 100,000-token knowledge base (about 75,000 words, or 125 average help articles). Every customer question goes to Claude Sonnet 4.6 at $3 per 1M input tokens and $15 per 1M output tokens.

Long context, no cache:

  • Input per question: 100,000 (knowledge base) + 500 (system prompt) + 200 (question) = 100,700 tokens
  • Output per question: 300 tokens
  • Cost: 100,700 × $3 / 1M + 300 × $15 / 1M = $0.3021 + $0.0045 = $0.31

At 10,000 questions a month: about $3,070. At 50,000: about $15,330.

RAG:

  • Retrieve 8 passages of 500 tokens = 4,000 tokens
  • Input per question: 4,000 + 500 + 200 = 4,700 tokens
  • Output: 300 tokens
  • Cost: 4,700 × $3 / 1M + 300 × $15 / 1M = $0.0141 + $0.0045 = $0.019

At 10,000 questions a month: about $186, plus the search index (pgvector on an existing database adds little; hosted vector databases charge separately). At 50,000: about $930.

On these assumptions, RAG costs about one-sixteenth as much per question ($0.3066 ÷ $0.0186 ≈ 16.5).

Prompt caching narrows the gap. On Sonnet 4.6, a cache hit costs $0.30 per 1M tokens, one-tenth of the base input price. If the 100,000-token knowledge base is a cached prefix:

  • Cached read: 100,000 × $0.30 / 1M = $0.03
  • Uncached system prompt and question: 700 × $3 / 1M = $0.0021
  • Output: $0.0045
  • Total: about $0.037 per question, roughly twice the RAG cost

Cache writes cost extra: $3.75 per 1M tokens for a 5-minute cache on Sonnet 4.6 ($0.375 to write the 100,000-token prefix), or $6 per 1M for a 1-hour cache. If traffic is sparse and the cache expires between questions, you pay the write again.

Latency

Time to first token is what a chat user feels.

RAG: a search query plus a model call with a short prompt.

Long context: no search step, but the model has to process a much larger prompt before it writes anything, and time to first token grows with prompt size. Caching helps here too, because cached tokens don't need to be processed again.

We have not published latency benchmarks, and results vary by region, load and tier. Measure time to first token on your own prompts at the sizes you plan to use. For a chat widget on a website, a delay of a few seconds before the first word is noticeable. For an internal tool where a person expects to wait, it matters less.

The accuracy question

The intuition that "more context is better" is only partly true.

The 2024 paper "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach" (Li et al., EMNLP 2024 industry track) compared both approaches on nine datasets with Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo. Its findings:

  • When sufficiently resourced, long context "consistently outperforms RAG in terms of average performance."
  • RAG's "significantly lower cost remains a distinct advantage."
  • Its Self-Route method, which sends each query to RAG or long context based on the model's own assessment, kept performance close to long context while cutting cost by 65% for Gemini-1.5-Pro and 39% for GPT-4O.

Those models are older than today's, so treat the direction as useful and the exact numbers as historical. Anthropic's own documentation adds a caution: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." It recommends curating what goes into context rather than filling it.

The practical read: long context can be slightly more accurate if the knowledge base fits and you can absorb the cost. RAG is cheaper and accurate enough for most support questions, provided retrieval finds the right passage. Test both on your own questions.

Side by side

DimensionRAGLong context
Cost per question (100k-token KB, Sonnet 4.6)about $0.019about $0.31, or about $0.037 on cache hits
Time to first tokenSearch plus a short promptGrows with prompt size; caching helps
Knowledge base size limitPractically noneThe model's context window (200k to 1M tokens above)
Updating contentRe-index the changed articleRebuild the prompt; the cache resets
SetupIndex, chunking, retrieval logicPaste the knowledge base into the prompt
Several customers or brandsFilter by account in the searchA separate prompt per account
Fits bestHigh volume, large or changing knowledge baseSmall knowledge base, low volume, prototypes
Typical failureRetrieves the wrong passageMisses a relevant fact buried in a long prompt

When long context wins

Small knowledge bases. If your help center is 80 articles of about 500 words (40,000 words, roughly 53,000 tokens), it fits any modern model. With Sonnet 4.6 and a warm cache, that is about $0.016 for the cached read plus $0.0045 output and a little uncached input: around $0.02 per question. Running a retrieval pipeline may not be worth it.

Conversation memory. Keeping the current conversation in the prompt is the natural way to give the model continuity. Retrieving over a customer's own recent messages adds complexity for little gain.

Prototypes. Before you build retrieval, paste your knowledge base into a prompt and see whether the answers are useful at all.

Long structured documents. A 50-page policy, an API spec or a contract that must be read as a whole. Chunking can split cross-references; long context keeps them together.

Low-volume internal tools. At 100 questions a day and $0.31 each, that is about $31 a day, or roughly $930 over 30 days. That may beat the engineering time for retrieval.

When RAG wins

Large knowledge bases. Once the content passes the model's window, long context is not an option, and near the limit both cost and the risk of missed facts rise.

High volume. At 10,000 or more questions a month, the roughly 16-to-1 cost ratio in the example above dominates.

Frequent updates. If content changes daily (product info, release notes, policies), RAG re-indexes one article. Long context rebuilds the prompt and pays for a new cache write.

Many tenants. Serving many customer accounts, each with its own knowledge base, is far simpler with filtered retrieval than with one giant prompt per account.

Thin margins per answer. Anything priced per answer or per resolution needs the cheaper path.

The hybrid most production systems use

Most support systems combine the two:

  • Conversation history in the prompt. The recent turns of the current chat go with each question.
  • Knowledge base through retrieval. Help articles, docs and policies are indexed, and the most relevant passages are retrieved per question.
  • Account data through tools. Subscription, order or ticket data is fetched on demand by a tool call, not stuffed into the prompt.
  • Whole documents when needed. When a question needs a full policy, the system can fetch that one document.

How Chatsy handles it: Chatsy agents retrieve passages from the sources you add (web pages, files, written answers and help-center articles) for each question and send them with the recent conversation, rather than placing your full knowledge base in every prompt. Account data is only available through custom API tools you build and test; connected services such as Stripe or Slack cannot run agent actions yet. Chatsy does not bill tokens: each AI response uses a fixed number of credits set by the model, from 1 to 30 depending on the model and plan (Claude Sonnet 4.6 is 3 credits on Hobby and above).

When this analysis changes

Cheaper models, with a threshold. Claude Haiku 5.5 lists $0.10 per 1M input tokens and $0.50 per 1M output tokens, but only for prompts up to 100,000 tokens. Above that, Anthropic charges $0.50 input and $2.50 output. Our 100,700-token long-context prompt crosses the line: 100,700 × $0.50 / 1M + 300 × $2.50 / 1M ≈ $0.051 per question. Trim the knowledge base to 99,000 tokens and the same question costs about $0.010. RAG on Haiku 5.5 stays under the threshold at about $0.0006 per question (4,700 × $0.10 / 1M + 300 × $0.50 / 1M), roughly 80 times cheaper than the long-context version. Compare like with like, and check that the small model's answers pass review.

Better caching. Anthropic already offers 1-hour caches, and its newest models (Claude Sonnet 5.5 and Opus 5.5) list cache hits at 0.05× the base input price. As cache prices fall, long context gets more competitive for stable content.

A tiny knowledge base. If you have 30 articles, none of this matters. Put them in the prompt and spend your time on the content.

Stick with RAG if

  • Your knowledge base has hundreds of articles or more.
  • You handle more than about 10,000 questions a month.
  • Your content changes daily.
  • You serve several customers or brands with separate knowledge bases.
  • Your budget per answer is a few cents or less.

If three or more apply, don't let a large context window talk you out of retrieval.

FAQ

Is RAG dead in 2026?

No. RAG is still cheaper and scales better than long context for most production support. What changed is the configuration: prompt caching, hybrid keyword and vector retrieval, and better embeddings. The core idea (retrieve, then generate) remains the default at meaningful volume.

Can I use a 1M-token model and skip RAG?

Technically yes. With Claude Sonnet 4.6 at $3 per 1M input tokens, a full 1M-token prompt costs about $3.00 per question uncached, or about $0.30 on a cache hit plus output. At 10,000 questions a month, that is about $30,000 uncached or about $3,000 on cache hits. Anthropic's own docs warn that recall degrades as context grows.

Does prompt caching make long context cheap enough?

Closer, but not equal. In the example above, caching takes the long-context cost from about $0.31 to about $0.037 per question, against about $0.019 for RAG on the same model. If your team has little engineering time, the simpler long-context setup may be worth the difference at low volume.

What about long context for conversation history and RAG for the knowledge base?

That is the hybrid described above and the common production pattern. Recent turns stay in the prompt; the knowledge base sits behind retrieval.

What this means for picking a chatbot

If you are building from scratch, start simple: long context on a small knowledge base with one model. Add retrieval when cost, latency or size forces it. Our vector search explainer and hybrid search guide cover the retrieval side, and RAG vs fine-tuning covers the other common question.

If you are buying, ask vendors what they actually send to the model for each question and how they bill: tokens, AI replies or resolutions.

Chatsy retrieves from your sources per question and lets you pick the model. The Free plan includes 5 models and 40 credits a month. Hobby and Standard include 38 models, Pro 41 and Enterprise 42. Paid self-serve plans run $40 to $500 a month, and Enterprise is quoted. See pricing for the full breakdown.

All articles
  • AI & Technology

    AI Chatbot Prompt Injection Defenses in 2026: What Works, What Does Not

    OWASP ranks prompt injection as the top LLM risk. See layered defenses for a support chatbot, a 10-case test you can rerun, and where Chatsy stops.

    May 20, 2026 · 13 min read

  • AI & Technology

    AI Chatbot Evaluation and Testing: A Support Team Playbook for 2026

    How to evaluate a customer support chatbot before launch using a 25-question golden set, four scoring dimensions, and the right tooling stack.

    May 20, 2026 · 12 min read

Build an agent from your support content

Add approved help content, test its answers, and decide when a person takes over.

No credit card required