RAG Embeddings, Retrieval and Context: Compare the Options

A user asks whether contractors can claim overseas travel. One passage uses “contractor”; another calls the same group “non-employees.” Keyword and semantic search may find different parts of the answer. The retrieval pipeline must bring the useful passages together, remove stale or inaccessible material and leave the generator enough context to explain the rule.

This chapter follows file preparation and chunking in our end-to-end RAG guide. We will use that policy question to connect embedding choices, candidate selection, reranking and the final evidence budget. Apply the permission rules in section 5 to every search path described here.

1. Keep four different limits separate

An 8,192-token embedding input does not produce 8,192 dimensions. A generator with a much larger window does not increase the embedding model's limit. Token counts can differ between models, especially for Chinese, code and OCR text.

Keep a configuration record with model identifier, revision where available, dimensions, normalization, distance metric, tokenizer and ingestion version. Treat an embedding change as a data migration: rebuild or maintain a separate versioned index, compare results, then switch queries. Equal dimensions alone do not establish compatibility.

2. Choose embeddings by your documents and questions

An embedding maps text to a sequence of numbers used for similarity search. It can bring “contractor” closer to “non-employee,” but similarity alone does not establish eligibility or current policy. Test the actual languages, product codes, abbreviations and messy text your users submit before choosing among these options.

OpenAI documents 1,536 default dimensions for small and 3,072 for large, with an 8,192-token maximum input for each. Its dimensions parameter supports shortening vectors. Evaluate the shortened configuration separately; fewer numbers are a storage choice, not evidence of equal retrieval quality. OpenAI embeddings.

Voyage's current table lists 32,000-token inputs for voyage-4, 4-large and 4-lite, with 1,024 default dimensions and 256, 512 or 2,048 alternatives. The same page lists open-weight voyage-4-nano. Follow documented query/document input modes, and explicitly inspect truncation behavior. Voyage text embeddings.

BGE-M3's model card lists 1,024 dimensions and an 8,192-token sequence length, plus dense, sparse and multi-vector modes. Self-hosting does not remove the need to check its license, dependencies and deployment footprint. BGE-M3 model card.

For visual evidence, Voyage documents interleaved text and visual inputs in multimodal-3.5. Keep original page images and page references even when embedding them: retrieval of a screenshot still needs a readable explanation and a navigable citation. Multimodal documentation.

Our selection test: freeze one question set and compare relevant-evidence recall, supported answers, latency and total operating cost. Include English questions against Chinese documents and the reverse if that reflects real use. Do not substitute an English benchmark score for this test.

3. Pick the storage and retrieval service independently

Sources: pgvector, Qdrant hybrid queries, Azure hybrid search, OpenAI retrieval.

Check index constraints before choosing dimensions. For example, pgvector documents up to 2,000 dimensions for its vector HNSW/IVFFlat indexes and 4,000 for halfvec. That is an index-type constraint, not a universal limit on what PostgreSQL can store. A default 3,072-dimensional embedding therefore needs a compatible strategy, such as supported dimensionality reduction or half-precision indexing, followed by evaluation.

Raw float32 storage illustrates the scale: one million 1,024-dimensional vectors require about 4.096 GB for the numbers alone. Text, metadata, graph indexes, replicas and backups add overhead. This is arithmetic, not a database capacity quote. Avoid estimating a production bill from vector bytes alone.

4. Compare lexical, dense, hybrid and reranked retrieval

Reciprocal rank fusion (RRF) combines positions in ranked lists. This avoids treating a lexical score of 12 and a vector similarity of 0.8 as directly comparable numbers. A passage near the top of both lists gains support from both; a passage found by only one method can still enter the combined result. Azure uses RRF to merge text and vector results; Qdrant documents multi-stage fusion. OpenAI's retrieval guide also exposes hybrid weighting. These controls remain platform-specific.

Start with an illustrative experiment: retrieve 30 lexical and 30 dense candidates, deduplicate by chunk/version, rerank the union, then pack the most useful evidence. Compare this with dense-only and lexical-only runs. The numbers are tuning inputs, not universal defaults.

For a concrete policy example, imagine “Can contractors claim overseas travel?” Lexical search matches “contractor,” while dense search finds a section titled “Non-employee international expenses.” Suppose the 60 candidates contain 12 duplicate chunk IDs, leaving 48. A reranker puts eight passages first, but four overlap the same parent section. Expand and deduplicate those into five evidence blocks. If each block averages 600 tokens including its source label, the evidence uses about 3,000 tokens. These are invented workload numbers illustrating the accounting, not measured search results.

Inspect the selected blocks before generating: one should establish worker eligibility, another the overseas condition, and another any approval exception. If none establishes eligibility, a high similarity score is insufficient. Return the missing-evidence response or perform another bounded search. This inspection gives a specific target for debugging: the missing fact, rather than an abstract “bad answer.”

A reranker examines candidates already retrieved; it does not search the missing parts of the corpus. If the eligibility passage never entered the candidate set, improve that search. If it entered but was discarded, inspect ranking and evidence packing. If it reached the generator but was ignored, investigate the answer stage. Log IDs at each boundary so those cases are distinguishable.

Voyage's page lists rerank-2.5 with a 32,000-token query-plus-single-document limit and separate query and batch constraints. It lists rerank-3 and 3-lite as preview. Do not pass an entire retrieved collection as one document because its generator context would fit. Reranker limits.

5. Enforce permission and version boundaries first

Derive tenant, user and group filters from authenticated server-side identity. Do not let the model supply an unrestricted tenant identifier. Apply the authorization constraint to every lexical branch, vector branch, parent expansion and document download.

Azure distinguishes preFilter, postFilter and strictPostFilter behavior; restrictive filtering after candidate selection can lose relevant matches. Its documentation recommends preFilter where recall is important. Vector filter behavior.

Our implementation recommendation is broader than one API setting: unauthorized content must never reach a reranker, generator, user-visible debug panel or cache shared across users. Also revalidate access when opening a citation. A search filter does not repair stale document permissions or an unprotected download URL.

Carry document version, effective date and status with each hit. For “current policy,” exclude superseded material through an explicit rule. For “what changed,” intentionally retrieve both versions and label them. Similarity cannot decide which version is legally or operationally applicable.

6. Rewrite queries carefully and expand useful context

A follow-up such as “Does it apply overseas?” needs the previous topic. Rewrite it into a standalone search request while preserving the original question. Keep exact identifiers, negatives, dates and units intact. Retain both queries in the trace so an incorrect rewrite is visible.

Try original-query retrieval alongside a rewrite before adding many generated variants. For a multi-part request, split only into meaningful subquestions. Each additional branch consumes latency and may introduce unrelated evidence.

For parent retrieval, search small chunks but load the surrounding section after a match. Store parent IDs during ingestion. Expand only when a definition, table header or exception is necessary; deduplicate repeated parents and recheck permissions. Fetching the entire manual for every matching sentence defeats the selection step.

7. Pack an explicit evidence budget

After selection and any parent expansion, measure what you will actually send. Use this planning equation, adapting it to the selected API's token accounting:

Evidence allowance = context ceiling − instructions/tools − question/history − output/reasoning reserve − safety margin.

An illustrative 32,000-token budget could reserve 3,000 for instructions and tools, 2,000 for question/history, 4,000 for response/reasoning, and 3,000 as margin. That leaves 20,000 for evidence. This is a worked example, not a model specification or advice to fill every available token.

Pack evidence by question coverage, relevance and authority. Preserve contradictory passages where they matter. Remove duplicate overlaps, repeated navigation and stale copies. Include a compact source label, document version and page/section locator with each excerpt. Count the final serialized request, including wrappers, rather than multiplying chunk count by an assumed size.

The policy example's five evidence blocks use about 3,000 tokens; a 20,000-token allowance is headroom, not a target to fill. If a required table cannot fit, retrieve its relevant rows with column headings or split the question. Preserve the exception that determines the answer before adding more loosely related passages.

8. Require supported claims and a useful no-answer path

Assign source IDs before generation. Ask for citations using those IDs, then validate that each cited source was actually supplied and is accessible to the user. Showing a citation does not prove the sentence follows from it; evaluate claim support separately.

A no-answer response should identify the missing evidence: “The supplied policy covers local employees; I found no overseas section.” It can offer a clarifying question or an authorized next search. Never manufacture a policy because the nearest document sounds relevant.

Treat retrieved text as evidence, including when it contains commands addressed to an assistant. It must not change tool permissions or override the application's instructions. Test malicious document passages as well as ordinary unanswered questions.

9. Decide whether advanced RAG earns its complexity

Microsoft GraphRAG builds entity relationships and community summaries, with different query modes for local and broader questions. That creates additional derived data to refresh and authorize. It does not justify converting every FAQ collection into a graph. GraphRAG documentation.

Azure's current agentic retrieval documentation separates generally available minimal extractive retrieval from preview capabilities such as LLM query planning and answer synthesis. Check the specific API version and capability, not only the product label. Agentic retrieval availability.

Our recommendation: keep the simplest pipeline that passes your evidence and permission tests. Add a stage to fix a measured failure, then rerun the same question set. Continue with the complete RAG workflow for evaluation and maintenance.