Latency and performance

Most of a reply's time is the model provider's own time to first token. SentientOne's share is small, and the parts it controls are cached or run in parallel. These are medians (p50) from our latest benchmark, run in October 2026; your model, prompt and network change them.

What keeps it fast

• Answer cache. A new conversation's first question, asked again to an unchanged agent without live tools, is answered from a 15-minute cache. It still costs 1 credit. See Chat stream.
• Retrieval caches. Query embeddings are cached for 7 days and search results for 5 minutes per agent; any knowledge change clears them.
• Hedged calls. When the embedding takes longer than 300 ms or the rerank longer than 400 ms, a second identical request goes out and the first good answer wins.
• Prompt caching on Anthropic. System prompts are marked for Anthropic's prompt cache. Other providers apply their own automatic caching, if any.
• Retries. Overloaded or unavailable provider responses (503, 529) are retried with backoff, up to three attempts. Authentication errors are never retried.
• History limits. Only the last 12 messages (up to 12,000 characters) go to the model with each message.
• Time limits. A whole reply has 120 seconds and each tool call 10 seconds; past them you get a TIMEOUT error event instead of a hung connection.

Time your own requests

Every response carries a Server-Timing header with the time spent in the API and upstream. Send X-Perf-Timing: 1 and the done event also carries a timing object with the stages of that reply. Send your own X-Request-Id to find the request in Tracing, where every span has its duration. See Request ids and Trace reference.

curl -N -D - https://api.sentientone.ai/v1/chat/stream \  -H "Content-Type: application/json" \  -H "X-Api-Key: $SENTIENTONE_API_KEY" \  -H "X-Agent-Id: $AGENT_ID" \  -H "X-Perf-Timing: 1" \  -d '{"message": "Where is order 10482?"}'

Making replies faster

• Choose a faster model for the agent. See Choose a model.
• Keep the system prompt and tool descriptions short; they're sent with every message.
• Answer common questions with exact Q&A pairs; their first asking is fast, and repeats hit the answer cache.
• Give MCP tools fast backends: a tool round adds the tool's time and a second model call.