LLM Gateway

Overview

LLM Gateway is an open-source API gateway for large language models. It sits between an application and model providers, exposing an OpenAI-compatible interface so developers can route requests across 250+ models and 40+ providers by changing a base URL rather than rewriting code. The project states it has routed over one trillion tokens and 80 million requests.

Built-in features include routing, caching, automatic failover, spend caps per key and member, and cost, latency, error and cache analytics. When a provider errors, times out or rate-limits, the gateway can retry on another provider within the same request. Enterprise adds SSO with SAML 2.0 and SCIM, role-based access, audit logs, guardrails that detect prompt injection and PII, and a 99.9% SLA. The platform states SOC 2 Type II compliance.

LLM Gateway can run in the vendor's cloud or be self-hosted with Docker or Kubernetes; the core is licensed under AGPLv3 and the enterprise code is source-available. Bringing your own provider keys is free, with a 5% fee on credit purchases; Enterprise pricing is custom.