The cost of a conversation depends on quoted prices, cache reads, and output tokens. If we are latency sensitive, there are tiers impacting this cost too. So the quoted price alone does not tell you what the next request will cost.
Kottos AI intelligent routing is derived from predictions generated from served requests. It accounts for cache behavior, price, timing, and venue health. The specific routing behavior is defined by the customer's preference.
Prompt caching is a property offered by some venues in which the most recent prompt is saved. Then, if the next prompt is just an extension of the previous one, most of the inference work is reused from the cached prompt. For instance, this is what happens in a Claude Code conversation where each question we ask is appended to the previous one so the model can answer within a context. Prices for cache reads are much cheaper than regular input token prices. See for Fable 5.1: reads at $0.25 vs inputs at $10.0. This is why our Fable 5.1 usage reports a blended $0.54 / M Token (99% cached). If a venue offers a lower input token price but no prompt caching, we need to account for the conversation effective net cost based on usage properties.
Therefore, deciding on a provider for a task is not as easy as it seems. Additionally, the choice could depend on tiers and venue health. So routing should be based on the executions we observe in the tape at the venue level.
A price-weighted rule assigns a probability to venues to decide when each will execute a request. Then, as requests come, they are sent randomly to each venue based on these probabilities. We simulated this routing model with two venues sharing the same quote price on 6,361 requests from our own traffic. Under this simulation, the token costs rose from $1,360 to $1,646, a 21% increase.
This simulation highlights the cost of losing cache for a specific task. It does not establish a universal routing penalty. The specific numbers will depend on usage, cache lifetime, and provider behavior.
When Kottos AI routes based on expected cost, its intelligence comes from predictions of specific usage. To verify these predictions, we store EXPECTED and ACTUAL request costs. Kottos AI never switches providers if the gain is below a config threshold. For other questions such as timing performance or whether a provider config choice was correct, we run simulations on the tape.
We keep track of provider quotes. These are stored in the tape for posterior cost adjustments if needed.
Use the live hosted product now. Sign up for an account and start with $10 in Kottos AI credits.
Observe what Kottos AI expects your request will cost which is what can be used to route.
Dedicated service for teams that need to shape provider integrations, deployment and routing to meet their goals.
We never log or store prompt/response content in the tape. Credentials are also only used to serve the request, never stored. Enterprise design partnerships cover metadata, provider access, deployment and workload-specific requirements.
Any deployment runs on llmbridge, our open-source C++ gateway under Apache 2.0. The Kottos AI layer with routing intelligence is built on top of it. The operational efficiency of llmbridge is provided in the benchmarks, and routing quality is evaluated on the tape.
Start with the live self-serve product, or talk with us about a dedicated enterprise service.