Knowledge and RAG
How retrieval works: hybrid search, fusion and rerank
What happens between a question and the passages the model sees: pgvector and full-text search, RRF fusion, Cohere rerank, thresholds and caches.
Concept2 min readUpdated 23 October 2026
Vector search is good at meaning and paraphrase but weak on exact strings: product codes, error codes, names, clause numbers. Keyword search is the opposite. SentientOne runs both on every question, merges them and lets a reranker decide.
The pipeline
- 1
Decide whether to search
Every question is searched up front, except obvious small talk such as greetings and thanks.
- 2
Embed the question
With OpenAI
text-embedding-3-small(1536 dimensions). Embeddings of repeated questions are cached for 7 days. - 3
Run two searches
A vector search with pgvector (cosine distance, HNSW index) and a Postgres full-text search (
plainto_tsquerywith the English configuration over a GIN index). Each returns up to 96 matches, filtered to this agent's chunks. - 4
Fuse
The two ranked lists are merged with reciprocal rank fusion (k = 60, equal weights), keeping the best 24 candidates.
- 5
Rerank
Cohere
rerank-v3.5scores each candidate against the question and the top 5 are kept. If the reranker doesn't answer within 1.5 seconds, or fails, the fused order is used, so a slow reranker never stalls a reply. - 6
Add context
If the best passage scores at least 0.30 (0.50 when the agent also has MCP tools), the passages are added to the question with an instruction to use them and cite documents and web pages. A
sourcesevent tells your app which ones.
Agents with knowledge also get a built-in search_documents tool, so the model can search again mid-answer, for example after a follow-up. Those calls show in the stream and in Tracing like any other tool call.
The numbers
| Setting | Value |
|---|---|
| Chunk size | About 400 words, 40 words of overlap |
| Embedding model | OpenAI text-embedding-3-small, 1536 dimensions |
| Vector index | HNSW, vector_cosine_ops |
| Full-text index | GIN over a stored tsvector, English |
| Matches per search | Up to 96 |
| Fusion | Reciprocal rank fusion, k = 60 |
| Candidates to rerank | 24 |
| Reranker | Cohere rerank-v3.5, 1.5 s timeout |
| Passages to the model | 5 |
| Minimum score to add context | 0.30, or 0.50 with MCP tools |
Fallbacks
- If the hybrid search fails, SentientOne falls back to vector-only search.
- If reranking times out or errors, the fused order is used.
- If nothing clears the minimum score, no context is added and the model answers from the prompt (and its tools).
Caches
| Cache | Lifetime | Cleared when |
|---|---|---|
| Question embeddings | 7 days | Never needed: the same text gives the same vector |
| Search results | 5 minutes | The agent's knowledge changes |
| Answers | 15 minutes | Any knowledge is added, or the agent's prompt, model or settings change |
The answer cache only serves the first question of a new conversation, and only for agents without live MCP tools. It's skipped when the request continues a conversation, overrides the prompt or sends X-Llm-Key. A cached answer arrives as a single delta with cached: true on the done event, and still costs 1 credit.
To see what was retrieved for a real question, open its trace: the rag.prefetch span holds the scores and the number of sources. See Debug a bad answer.
Still stuck?
Email us with what you tried and any request id. A person replies.