llmbridge is a drop-in, OpenAI-compatible LLM gateway written in C++. It translates OpenAI requests to and from Anthropic, Gemini, and Cohere dialects with microsecond overhead and zero runtime dependencies. It is open source under Apache 2.0.

How much latency does llmbridge add?

About 0.08 ms at p99 and 100 RPS on its own harness, holding 41 to 80 microseconds p99 from 100 to 5,000 RPS, against a real request whose time to first token is around a second. On the independent ENTERPILOT AI gateway reproducible benchmark it adds 0.07 ms p50 on chat completions and 0.18 ms streaming, the same as a socat byte pipe, on one CPU core; GoModel adds 0.47 ms on six cores, Bifrost 0.80 ms on eight, Portkey 7.6 ms and LiteLLM 10.8 ms. A single-threaded worker reaches about 84,000 requests per second. On streaming workloads it keeps time to first token on the no-gateway floor of about 31 ms through 512 concurrent streams while delivering 99.93 to 100 percent of tokens.

Does llmbridge have runtime dependencies?

No. llmbridge is a self-contained C++20 binary with zero third-party runtime dependencies, including a hand-rolled JSON parser scoped to chat-completion payload shapes.

Is llmbridge free and open source?

Yes. llmbridge is free and open source under the Apache 2.0 license. Kottos AI also offers a hosted gateway built on the same engine, adding real-time intelligence that establishes best price-performance, observability, and managed infrastructure.

Does llmbridge support streaming (SSE)?

Yes. llmbridge translates a live Anthropic server-sent-events stream into OpenAI chat.completion.chunk events token by token. Streaming runs on both event-loop backends (io_uring and epoll), applies back-pressure when the client is slower than the provider, bounds stalled upstreams with an idle timeout, and emits the final usage chunk when the request sets stream_options.include_usage. Tool calls stream too: Anthropic's tool_use and input_json_delta events become OpenAI tool_calls deltas, so an agent loop works over SSE. Streaming for the Gemini and Cohere dialects is not implemented yet.

Kottos AI: exchange-grade infrastructure for LLM inference

Q: Isn't the model the bottleneck, not the gateway?

For a single request, yes, and we say so first: the model takes about a second and the gateway is 0.005 percent of it. What the overhead buys is headroom. One worker carries about 84,000 requests per second and 24,576 concurrent streams, so the same volume runs on a fraction of the hardware, and there is no garbage collector to spike the tail on a real-time budget like voice. The latency that is worth attacking is not the gateway at all, it is which venue serves the request.

gateway	runtime	non-streaming	streaming	peak req/s	cores used	req/s per 1% CPU
llmbridge v0.53.0 ours, Kottos AI	C++	0.07 ms	0.18 ms	40,677	0.98	354
GoModel 0.1.86	Go	0.47 ms	0.75 ms	16,468	5.96	23
Bifrost 2.0.0	Go	0.80 ms	2.77 ms	10,043	7.90	11
Portkey 1.15.2	Node	7.56 ms	27.60 ms	1,232	1.20	10
LiteLLM 1.99.1	Python	10.77 ms	46.06 ms	880	7.05	1
socat byte pipe the floor	C	0.06 ms	0.27 ms	43,973	2.17	157

01 / The problem

Two things your gateway costs you.

The milliseconds it adds to every request, and the venue it picks for you. The first is measurable to the microsecond. The second shows up on the bill, and only a per-request record shows you where.

// latency

The gateway in front of you

LiteLLM, the default choice, is Python: on an independent harness it adds 11 ms at the median per request and 46 ms streamed, on seven cores, and caps out at a few hundred requests per second per worker. Voice agents have ~300 ms turn budgets; agentic workflows chain 5–30× more calls per task than a chatbot. The overhead compounds on exactly the workloads growing fastest.

// the bill

Routing on posted price, blind to your cache

A router that picks the venue on the quoted $/M ignores what the request actually costs you. A prompt cache lives at one venue: switch to the one that looks cheaper and the cache misses, so the $0.25 cache read becomes a $10 input token (Claude Fable 5.1, direct). OpenRouter's router chose Bedrock for us at +251 ms median over the pinned venue (our measurement, Aug 2026), and its default weighting is by posted price alone. The quoted price and the paid price part ways, and nothing on a price page tells you by how much. Replayed on 6,361 requests of our own traffic, that rule cost 21% more for no price gain (how we measured it).

02 / Proof

The gateway adds almost nothing.

The external table above is one harness we did not write. These two are ours: llmbridge against LiteLLM doing the same OpenAI↔Anthropic translation, request by request and then streamed. The full set, the sweep to 24,576 concurrent streams, the stamp-by-stamp timeline of one request, the methodology and the raw data are on the benchmarks page.

Added p99 per request, one worker each, same translation work against the same mock. LiteLLM 1.95.0 pinned; ~1,000× at 100 RPS.

Streaming: time to first token vs concurrent streams

Streaming time to first token the gateway adds, swept over concurrent streams. llmbridge stays on the no-gateway floor through 512; LiteLLM delivers 4% of the tokens there.

// common question

Isn't the model the bottleneck, not the gateway?

For a single request, yes, generation takes seconds and the gateway is invisible. The efficiency shows up at scale: at millions of requests a day, LiteLLM's ~250 RPS per worker means dozens of processes burning CPU and memory, while llmbridge handles the same volume on a fraction of the hardware. It compounds in agentic workloads (dozens of calls per task) and on real-time budgets like voice, which is exactly what the streaming benchmark above shows: at 512 concurrent streams LiteLLM's first token arrives after ~12.6 seconds and 96% of the tokens never arrive at all, while llmbridge stays on the no-gateway floor. And most of all, it's what powers best price-performance across providers (the same model is priced up to ~10× apart, an estimated 15–40% lower cost) without adding latency of its own.

All the benchmarks and the methodology → Raw data on GitHub

03 / Architecture

Built the way a trading system would be.

No GC, no locks on the hot path, no third-party runtime dependencies. Designed by someone who spent nine years in high-frequency and electronic trading, where latency budgets and tail distributions are the job.

// event loop

Single-threaded io_uring

Completion-driven I/O (multishot accept/recv, provided buffers) with a keep-alive upstream pool, and epoll fallback for older kernels. One core sustains ~84k RPS, and scales out with SO_REUSEPORT.

// zero deps

Hand-rolled everything

A tiny dependency-free JSON parser/serializer scoped to the chat-completion shapes, and zero-copy string_view over input buffers.

// deterministic

No GC pauses

It's C++. Tail latency is bounded by malloc, not garbage collection, so p99.9 stays flat under load.

The flags

#  --upstream-dialect openai|anthropic|gemini|cohere|bedrock|azure   --io auto|epoll|uring   --workers N
#  --upstream-timeout SECONDS   (upstream silence before a request/stream is aborted)
#  TLS on either leg needs a TLS build (-DLLMBRIDGE_TLS=ON); the default build is dependency-free.
#  --listen-tls --tls-cert CERT --tls-key KEY terminates the client's TLS. --config FILE names an upstream table.

Prefer a library? Call the translation functions directly from C++: llmbridge::provider::openai_to_anthropic_request(...) and back. Same zero-dependency core.

04 / Open source

Open source at the core

llmbridge is the fast gateway core, free under Apache 2.0. We maintain it; you use it without restriction.

Available now v0.54.1 · beta

Chat completions, drop-in OpenAI-compatible
SSE streaming (token-by-token) for OpenAI ↔ Anthropic, with back-pressure and usage passthrough
OpenAI ↔ Anthropic, Gemini and Cohere translation; AWS Bedrock (SigV4) and Azure OpenAI as venues
Byte-forward for OpenAI-compatible providers (Groq, Together, Fireworks, DeepInfra, Mistral, ...) and for Anthropic clients on an Anthropic venue
Tool calling: declarations, tool_choice, parallel calls, tool_result round-trip, streaming and non-streaming, and cache_control forwarded byte for byte
TLS on both legs: to real providers with credential passthrough, and --listen-tls for the client’s connection; verification is always on
An upstream table with per-request venue selection through a policy seam, and a failure hook that can re-dispatch a request
Per-request timing headers (gateway vs connect vs provider), time to first token and first thinking delta, the provider’s own token counts, and a request sink for a tape
Zero runtime dependencies · Linux · io_uring (epoll fallback)

On the roadmap next

Bedrock streaming: the AWS event-stream decoder, so a streamed request to Bedrock is served instead of refused
An Anthropic-speaking client to a Bedrock venue, which is what makes cross-venue routing for Claude reachable from Claude Code and the Anthropic SDKs
Vision / multimodal, Anthropic-in translation (an Anthropic client on an OpenAI venue), and streaming for Gemini / Cohere
Python / Go / Rust bindings

View llmbridge on GitHub ↗

05 / The commercial layer

Run it hosted, or run it yourself.

The paid product is the intelligence, not the hosting. Two ways to take it: hosted, i.e. point your client at our endpoint and we operate the data plane; or self-hosted, the same container in your own VPC, so prompts and provider keys never leave your network, connected to our control plane for the routing intelligence. Same engine and same intelligence either way; the only difference is where the data plane runs. And llmbridge itself stays free under Apache 2.0 whichever you choose.

Best price-performance

The price book runs live today: the same model tracked across providers in real time (see the ~10× spread below). The latency/error book is measured on every hosted request: a rolling p95 per venue that routing consults. The cross-provider comparison is in build with design partners: venue selection routed by us when we host, recommended to your gateway when you host. At-cost token passthrough and a separate charge for the routing, never a percentage of token spend and never a token markup.

Production observability

Per-request tracing, token-cost attribution by team/project, budgets, and audit logs. Anonymized routing metadata only. prompt content is never logged.

Founder support

You work with us directly. Drop-in migration from LiteLLM/OpenRouter/Portkey: change your base URL and key, existing code works.

Private beta. hosted by us or as a container inside your own VPC, same engine and same intelligence, prompts never leaving your network. The hosted gateway serves its first customers today at api.kottos.ai, with every request on the tape. See what's included and apply →

06 / Market

One model, many venues, one market.

The same model (say Llama-3.3-70B) is served by 15+ providers at up to ~10× price dispersion and 5–20× latency variance. That's a commodity market structure. We think it should be priced and executed like one. The people who execute in a commodity market best are the ones who've spent years pricing, executing, and measuring latency inside one.

Llama 3.3 70B · price per 1M tokens · same model, every provider loading live prices...

Provider	Input $/M	Output $/M	Quant	Context	Uptime
Fetching live prices from OpenRouter...

Live from OpenRouter when this page loads: the same model, priced up to ~10× apart across providers (Llama-3.3-70B today). But they differ on quantization, context, and uptime (columns at right), so the cheapest is often the quantized / smaller-context one. Routing captures the gap among providers that meet your SLA, not the cheapest overall. See the hosted gateway →

Inference has ratings agencies for the issuer and no tape for the fill.

Evals grade the model; nobody publishes what a given venue delivered on a given request. Corporate bonds traded that way until TRACE made the fills public in 2002, and spreads compressed without anyone adding a bid side. The full thesis: how inference trades, the inference map, and where the analogy breaks →

Kottos AI is building exchange-grade infrastructure for LLM inference: an open-source microsecond gateway today, real-time intelligence that establishes best price-performance next, and (as GPU compute financializes) price discovery for inference capacity itself. The open core is how you start; the routing intelligence is where it compounds.

Lluís Antoni Jiménez Rugama

Founder. PhD in applied mathematics, then a decade in finance, nine years of it in high-frequency and electronic trading (Virtu, UBS, with Société Générale before them) building intuition for latency, market microstructure, and execution quality, with machine-learning work along the way. llmbridge is that same instinct, applied to routing inference.

LinkedIn ↗ · GitHub · [email protected]

Exchange-grade
infrastructure for
LLM inference.

Drop it in