llmbridge is a drop-in, OpenAI-compatible LLM gateway written in C++. It translates OpenAI requests to and from Anthropic, Gemini, and Cohere dialects with microsecond overhead and zero runtime dependencies. It is open source under Apache 2.0.

How fast is llmbridge compared to LiteLLM?

On an equal-work benchmark (both gateways doing the full OpenAI to Anthropic translation against the same mock backend), llmbridge adds about 0.067 ms of p99 latency at 100 RPS versus about 82 ms for LiteLLM — roughly 1,200× lower added p99 — and sustains about 90k RPS on a single core (loopback-bound on the test host, not CPU-bound).

Does llmbridge have runtime dependencies?

No. llmbridge is a self-contained C++20 binary with zero third-party runtime dependencies, including a hand-rolled JSON parser scoped to chat-completion payload shapes.

Is llmbridge free and open source?

Yes. llmbridge is free and open source under the Apache 2.0 license. Kottos AI also offers a hosted gateway built on the same engine, adding real-time multi-provider routing, observability, and managed infrastructure.

Kottos AI — exchange-grade infrastructure for LLM inference

Q: Isn't the model the bottleneck, not the gateway?

For a single request, yes — the model's generation takes seconds and the gateway is invisible. The gateway's efficiency matters at scale: at millions of requests per day, LiteLLM caps around 250 RPS per worker (so dozens of processes), while llmbridge handles the same volume on a fraction of the hardware. It also compounds in agentic workloads with dozens of calls per task, on real-time budgets like voice where garbage-collection pauses spike tail latency, and it is what lets routing across providers cut 15 to 40 percent of cost without adding latency of its own.

01 / Inference map

One model. A dozen providers. Spread across the country.

The same open-weights model — Llama, DeepSeek, Qwen — is served from datacenters coast to coast. Distance is latency: the Bay Area and Virginia are ~60 ms RTT apart on today's fiber, before the model emits a single token. Routing has to know the map.

Metro hubs are approximate; provider placement is illustrative of where open-weights inference clusters, not a claim about any one provider's exact facility. Distances are great-circle; the ~60 ms RTT is a real-world fiber estimate, not a Kottos measurement (cf. AWS N. California↔N. Virginia ≈ 58 ms) — the straight-line speed-of-light floor is ~26 ms. Node color is static today — live demand coloring (green→amber→red from real measured load) ships with the hosted gateway's shadow order book.

02 / The problem

Your LLM gateway is your bottleneck.

Voice agents have ~300 ms budgets. Agentic workflows chain 5–30× more calls per task than a chatbot. Every millisecond of routing overhead compounds — and the gateways that developers reach for were never built for these workloads.

LiteLLM

Python. Per-request object churn adds tens to hundreds of ms, and per-worker throughput caps out at a few hundred RPS.

Vendor gateways

Lock you to one provider. No real routing across the 15+ providers serving the same open-weight model.

OpenRouter

~5% markup and ~25 ms added per request. A switch with a margin, not a latency-first router.

03 / Benchmark

The benchmark

Equal work: both llmbridge and LiteLLM do the full OpenAI↔Anthropic translation against the same mock backend, driven by the same open-loop, coordinated-omission-corrected load generator. This measures gateway overhead, not the model's own latency. LiteLLM is a full router/SDK and does far more than translate — we compare only the gateway hop both perform.

llmbridge vs LiteLLM — added p99 latency

Throughput saturation — offered vs achieved RPS

llmbridge's ceiling here (~90k RPS) is the loopback's packet-processing limit, not the CPU's — the proxy uses about one core at saturation. Everything runs over 127.0.0.1, where the kernel's per-packet path saturates (~350–370k packets/s on this host) well before the gateway does. On real NICs with multiple workers there's substantial headroom; a separate-host run is on the roadmap.

RPS	llmbridge added p99	LiteLLM added p99	LiteLLM throughput
100	0.067 ms	81.77 ms	99 / 100 ✓
1000	0.078 ms	9,782 ms	245 / 1000 (sat)
5000	0.042 ms	12,438 ms	233 / 5000 (sat)

Honest methodology: single Linux host (i7-9750H), mock + proxy + load generator co-located, single worker/thread each, plain HTTP, mock backend (the model's latency is excluded on purpose, to isolate gateway overhead). llmbridge is proxy-self-measured; LiteLLM is client-measured (e2e − backend). The throughput ceiling on this box is loopback packet processing, not CPU — the proxy never exceeds ~1 core, so the limit is the 127.0.0.1 kernel path, not the gateway. Fully reproducible — ./bench/run_headtohead.sh and ./bench/saturate.sh. A clean separate-host run is on the roadmap.

// common question

Isn't the model the bottleneck, not the gateway?

For a single request, yes — generation takes seconds and the gateway is invisible. The efficiency shows up at scale: at millions of requests a day, LiteLLM's ~250 RPS per worker means dozens of processes burning CPU and memory, while llmbridge handles the same volume on a fraction of the hardware. It compounds in agentic workloads (dozens of calls per task), on real-time budgets like voice (where GC pauses spike the tail), and — most of all — it's what lets the hosted gateway route across providers for 15–40% lower cost without adding latency of its own.

04 / Architecture

Built the way a trading system would be.

No GC, no locks on the hot path, no third-party runtime dependencies. Designed by someone who spent a decade in high-frequency and electronic trading, where latency budgets and tail distributions are the job.

// event loop

Single-threaded io_uring

Completion-driven I/O (multishot accept/recv, provided buffers) with a keep-alive upstream pool; epoll fallback for older kernels. One core sustains ~90k RPS; scale out with SO_REUSEPORT.

// zero deps

Hand-rolled everything

A tiny dependency-free JSON parser/serializer scoped to the chat-completion shapes, and zero-copy string_view over input buffers.

// deterministic

No GC pauses

It's C++. Tail latency is bounded by malloc, not garbage collection — so p99.9 stays flat under load.

Drop it in

# Front an upstream (IPv4:port, plain HTTP in v0.1) and translate OpenAI ↔ provider on the fly:
llmbridge --listen 8088 --upstream 10.0.0.20:9001 --translate anthropic --workers 4
#         --translate none|anthropic|gemini|cohere      --io auto|epoll|uring      --workers N

# Clients keep speaking the OpenAI API — point them at :8088, nothing else changes.
# TLS and direct provider hostnames (api.anthropic.com, …) are on the roadmap.

Prefer a library? Call the translation functions directly from C++ — llmbridge::provider::openai_to_anthropic_request(...) and back. Same zero-dependency core.

05 / Open source

Open source at the core

llmbridge is the fast gateway core, free under Apache 2.0. We maintain it; you use it without restriction.

Available now v0.2.0 · alpha

Non-streaming chat completions, drop-in OpenAI-compatible
OpenAI ↔ Anthropic, Gemini, and Cohere translation
Passthrough for OpenAI-compatible providers (Groq, Together, Fireworks, DeepInfra, Mistral, …)
Zero runtime dependencies · 576 tests · Linux · io_uring (epoll fallback)

On the roadmap Phase B/C

SSE streaming (token-by-token) and tool calling
Vision / multimodal and cache_control
AWS Bedrock and reverse-direction translation
TLS, real provider endpoints, and Python / Go / Rust bindings

View llmbridge on GitHub ↗

06 / Hosted gateway

The hosted gateway

The same C++ engine, plus everything you'd otherwise build: real-time routing across providers, observability, team management, and managed infrastructure.

Intelligent routing

We measure every major provider's actual latency, throughput, and error rate in real time — and route around degradations before status pages update. At-cost passthrough pricing, no token markup.

Production observability

Per-request tracing, token-cost attribution by team/project, budgets, and audit logs. Anonymized routing metadata only — prompt content is never logged.

Founder support

You work with us directly. Drop-in migration from LiteLLM/OpenRouter/Portkey — change your base URL and key, existing code works.

07 / Thesis

Why us

The same model — say Llama-3.3-70B — is served by 15+ providers at up to ~10× price dispersion and 5–20× latency variance. That's a commodity market structure. We think gateways should route it like one. The people who route a commodity market best are the ones who've spent years pricing, executing, and measuring latency inside one.

Llama 3.3 70B · price per 1M tokens · same model, every provider loading live prices…

Provider	Input $/M	Output $/M	Quant	Context	Uptime
Fetching live prices from OpenRouter…

Live from OpenRouter when this page loads — the same model, priced up to ~10× apart across providers (Llama-3.3-70B today). But they differ on quantization, context, and uptime (columns at right), so the cheapest is often the quantized / smaller-context one. Routing captures the gap among providers that meet your SLA — not the cheapest overall. See the hosted gateway →

Kottos AI is building exchange-grade infrastructure for LLM inference: an open-source microsecond gateway today, a real-time routing layer next, and — as GPU compute financializes — price discovery for inference capacity itself. The open core is how you start; the routing intelligence is where it compounds.

Lluís Antoni Jiménez Rugama

Founder. A decade in high-frequency and electronic trading — Virtu, UBS, and Société Générale — building intuition for latency, market microstructure, and execution quality, with machine-learning work along the way. llmbridge is that same instinct, applied to routing inference.

LinkedIn ↗ · GitHub · [email protected]

Exchange-grade infrastructure for LLM inference.