The cost of a conversation depends on quoted prices, cache reads, and output tokens. If we are latency sensitive, there are tiers impacting this cost too. So the quoted price alone does not tell you what the next request will cost.
Kottos AI intelligent routing is derived from predictions generated from served requests. It accounts for cache behavior, price, timing, and venue health. The specific routing behavior is defined by the customer's preference.
Prompt caching is a property offered by some venues in which the most recent prompt is saved. Then, if the next prompt is just an extension of the previous one, most of the inference work is reused from the cached prompt. For instance, this is what happens in a Claude Code conversation where each question we ask is appended to the previous one so the model can answer within a context. Prices for cache reads are much cheaper than regular input token prices. See for Fable 5.1: reads at $0.25 vs inputs at $10.0. This is why our Fable 5.1 usage reports a blended $0.54 / M Token (99% cached). If a venue offers a lower input token price but no prompt caching, we need to account for the conversation effective net cost based on usage properties.
Therefore, deciding on a provider for a task is not as easy as it seems. Additionally, the choice could depend on tiers and venue health. So routing should be based on the executions we observe in the tape at the venue level.
A price-weighted rule assigns a probability to venues to decide when each will execute a request. Then, as requests come, they are sent randomly to each venue based on these probabilities. We simulated this routing model with two venues sharing the same quote price on 6,361 requests from our own traffic. Under this simulation, the token costs rose from $1,360 to $1,646, a 21% increase.
This simulation highlights the cost of losing cache for a specific task. It does not establish a universal routing penalty. The specific numbers will depend on usage, cache lifetime, and provider behavior.
When Kottos AI routes based on expected cost, its intelligence comes from predictions of specific usage. To verify these predictions, we store EXPECTED and ACTUAL request costs. Kottos AI never switches providers if the gain is below a config threshold. For other questions such as timing performance or whether a provider config choice was correct, we run simulations on the tape.
We keep track of provider quotes. These are stored in the tape for posterior cost adjustments if needed.
Start recoding your requests usage, timing and errors through Kottos, so you can begin to make changes in your configurations to save time or money.
Observe what Kottos AI expects your request will cost which is what can be used to route.
This is in development with design partners. You can tell us what providers you use so we can see what is supported today or what needs further integration.
The easiest deployment. Requests pass through our gateway and Kottos AI executes the routing decisions.
You can run the gateway in your server. In this case, Kottos AI only provides routing intelligence and you have the last word into where requests are sent. Kottos AI only sees routing and metadata usage.
We never log or store prompt/response content in the tape. Credentials are also only used to serve the request, never stored. Beta discussions cover: metadata collected, provider access, and deployment arrangements.
Any deployment runs on llmbridge, our open-source C++ gateway under Apache 2.0. The Kottos AI layer with routing intelligence is built on top of it. The operational efficiency of llmbridge is provided in the benchmarks, and routing quality is evaluated on the tape.
If you have suffered from any of the issues above or want to know more, you can try measuring your Claude Code requests, or reach us out to discuss a private pilot.