How to scale a RAG system

A RAG system does not have one scale problem. Uploads consume CPU and external API capacity. Retrieval depends on index size and query shape. Answer generation depends on model latency, token use, and tool calls.

Treat those paths separately. You can then measure the slow part, add capacity where it matters, and avoid copying an architecture built for somebody else's traffic.

In brief

• Move document processing out of the request path once uploads can exceed a normal HTTP request window.
• Keep every query scoped to one workspace before it reaches the vector index.
• Add caches only after you know which repeated work is expensive.
• Set capacity and alert thresholds from your own traces and load tests.

Start with a workload table

Before choosing workers, indexes, or caches, write down the workload you expect.

Use ranges, not one average. A ten-page guide and a scanned annual report place very different loads on the ingestion path.

Separate online and background work

The chat request path should contain only work needed for the current answer:

request -> authorize workspace -> retrieve -> generate or hand off -> stream response

Document parsing belongs in a queue:

upload -> create source record -> enqueue job -> parse -> chunk -> embed -> index

Return a source ID as soon as the upload is accepted. Let the interface show queued, processing, ready, or failed. Do not report a source as ready until its searchable chunks are committed.

Make jobs safe to retry

Workers crash. Providers time out. Deployments interrupt jobs. A retry must not create duplicate chunks or charge for embeddings you already stored.

Use a stable identity for each processing attempt:

const jobKey = `${workspaceId}:${sourceId}:${contentHash}`

Before expensive work, check whether that content hash already has a completed result. Write chunks and status in a transaction where practical. Keep the original source record so a failed run can show a useful error.

A worked capacity example

Suppose a local test shows that one representative document takes 30 seconds to parse, chunk, and embed. Five hundred documents would take 250 minutes on one serial worker:

500 documents × 30 seconds = 15,000 seconds = 250 minutes

That is a worked example, not a production benchmark. Your number will change with file type, chunk count, provider limits, and network conditions.

If ten workers can run without hitting database or embedding limits, the theoretical floor is 25 minutes. Real runs take longer because work is uneven and retries occur. Load test the complete queue before promising an import time.

Scope retrieval before tuning it

Every retrieval query needs a workspace or agent filter. This is a data boundary first and a performance filter second.

SELECT id, content
FROM knowledge_chunks
WHERE workspace_id = $1
ORDER BY embedding <=> $2
LIMIT $3;

Confirm the query plan uses the intended filter and vector index. Test with a large workspace and many small ones. A fast unscoped query is still wrong.

Choose the index from measurements

pgvector supports exact search and approximate indexes such as HNSW and IVFFlat. The right choice depends on recall requirements, write rate, memory, build time, and the number of vectors searched after filtering.

Use this sequence:

1. Keep an exact-search sample as the relevance reference.
2. Test candidate indexes on the same vectors and queries.
3. Compare recall, median latency, tail latency, memory, and index build time.
4. Repeat after the corpus grows or chunking changes.

A threshold copied from another system does not tell you where your index should switch.

Cache repeated work, not everything

Three cache candidates are common.

Embeddings. Key the cache by model, model version, normalization settings, and a hash of the exact text. Reusing an embedding from a different model silently corrupts retrieval.

Retrieval results. Include workspace, permissions, query, filters, and index version in the key. Invalidate when relevant source content changes.

Final answers. Cache only when the answer does not depend on account state, identity, time-sensitive policy, or a tool result. A shared answer cache can leak one customer's data to another if the authorization scope is missing from the key.

Measure hit rate and saved work for each cache. Remove a cache that adds invalidation bugs without reducing meaningful load.

Stream for feedback, not to hide slow work

Streaming lets the interface show the answer as the model produces it. It does not reduce retrieval time or the full generation time.

Track at least these two values separately:

• Time until the first visible token
• Time until the answer and citations are complete

If retrieval consumes most of the wait, streaming cannot fix it. Profile the request before changing the interface.

Route models only with an evaluation rule

A smaller model can handle some requests at lower cost, but a vague "simple versus complex" classifier is hard to audit. Define observable routing rules and test both routes.

Examples include source count, tool requirement, structured output requirement, and a labeled topic. Keep a fallback to human support for tasks where the cheaper route fails the acceptance set. This is a product rule, not an excuse to retry models until one sounds right.

Measure each stage

Aggregate API latency hides the cause. Record spans for:

• Authorization and rate limiting
• Query embedding
• Keyword and vector retrieval
• Reranking
• Model time to first token and completion
• Tool calls
• Queue wait and document processing

Break results down by workspace, file type, model, and outcome. Use percentiles rather than averages. Set alerts only after you have a normal baseline and an owner who can act on them.

Test failure, not only throughput

A useful load test includes provider timeouts, duplicate delivery, a worker restart, a database connection limit, and one unusually large source. Check that:

• Jobs resume without duplicate chunks
• Backpressure slows intake before dependencies collapse
• Failed sources show a clear status
• One large workspace does not block the rest
• The agent never searches content from another workspace

Throughput is only one part of a scalable system. Predictable failure matters more on a bad day.

When a simpler system is better

Keep synchronous ingestion and exact search if the corpus is small, updates are rare, and measured response times already meet the target. A queue, Redis, partitioning, and multiple index types create operational work. Add them when a trace or capacity test shows the need.

The scalable design is the one your team can explain, test, and repair. Start with the workload, not a diagram.

For the storage tradeoff, read Pinecone to pgvector migration guide. For retrieval, see hybrid search explained.

Frequently asked questions

When should document ingestion move to a queue?

Use a queue when parsing and embedding can outlast a normal request, when users upload batches, or when provider limits require controlled concurrency.

Does partitioning guarantee stable vector-search latency?

No. Filtering or partitioning can reduce the searched set, but latency still depends on index choice, data distribution, query settings, hardware, and concurrent load.

What should I cache first?

Profile the system. Reused embeddings are often a safe first candidate because the key can include the exact text and model version. Final-answer caching needs stricter authorization and freshness rules.

How do I set an ingestion target?

Run representative files through the full pipeline, include provider limits and retries, then choose a target the tested capacity can support with headroom.