LiteLLM vs agentgateway: comparing two open-source AI gateways

LiteLLM vs agentgateway looks like an even match on paper. Both are open source, both are self-hosted, both put an OpenAI-compatible endpoint in front of many model providers, and both now handle MCP tool calls and agent-to-agent traffic as well as model calls. They come from very different places, though. LiteLLM is a Python proxy from BerriAI that grew out of a popular SDK, and it has the biggest feature list and community in the category. agentgateway is a Rust proxy that Solo.io wrote from scratch and handed to the Linux Foundation, and it is smaller and faster, with a narrower focus. If you are looking at LiteLLM alternatives, it is one of the few that can match LiteLLM on agent features.

We have run both. Our team tested the open-source agentgateway build hands-on (v1.5.0, with the main findings re-checked on v1.6.0) and ran LiteLLM through the same benchmark setup. We then checked the claims in this post against each project's docs in November 2026, when the current releases were agentgateway v1.7.0 and LiteLLM v1.106.6.

LiteLLM: overview

LiteLLM started life inside BerriAI, a Y Combinator company, and was open-sourced in August 2023. Most people meet it as a Python SDK first and then run the LiteLLM proxy as a shared gateway. It now has more than 61,000 GitHub stars and public users including Netflix, Okta and NVIDIA. The docs list 100+ providers (the homepage says 140+), and you can route to self-hosted models like vLLM or Ollama alongside the commercial APIs.

The free version covers a lot. You get virtual keys, budgets and rate limits at the key, user, team, organization, end-user and tag level, plus exact-match and semantic caching, half a dozen routing strategies, fallbacks, Prometheus metrics and dozens of logging integrations. The LiteLLM MCP gateway can restrict access down to individual tools and even individual tool parameters, and an A2A agent gateway has shipped since v1.80.8.

The cost is in operations. The proxy is a Python (FastAPI) app that needs Postgres for keys, the admin UI and spend tracking, and Redis once you run more than one instance. LiteLLM is moving its hot path to Rust, and its homepage now advertises a Rust gateway, but the docs still call the Rust core a beta that covers a few routes, with the full Anthropic Messages path targeted for December 2026. Enterprise features such as SSO for more than five users, SCIM, audit logs and JWT authentication for workloads need a paid license.

agentgateway: overview

agentgateway is Solo.io's replacement for its older Gloo AI Gateway, written in Rust rather than built on Envoy. It is licensed Apache 2.0 and hosted by the Agentic AI Foundation, part of the Linux Foundation, and Solo sells a paid edition called Solo Enterprise for agentgateway. It runs as a single binary reading one YAML file, as a container, or on Kubernetes with its own controller, where it implements the Gateway API. It doesn't need a database to route traffic. Our team had it serving requests in under 11 seconds from a cold start, using about 18 MB of memory at idle.

It governs LLM calls, MCP traffic and A2A calls with one policy model. MCP authorization is its strongest feature: CEL rules decide which tools each caller can use, and the gateway filters the tool list so an agent only sees what it is allowed to call. The free edition also validates JWTs against your identity provider, limits models per key, enforces per-key budgets in the standalone build and masks sensitive data like email addresses and card numbers before a prompt reaches the provider. Fleet-wide rate limits, a shared budget resource and a cost management UI are enterprise features. Our agentgateway review covers it in depth.

Head-to-head comparison

The LiteLLM column reflects LiteLLM's docs and our own benchmark run. The agentgateway column reflects our hands-on testing and Solo's docs for v1.7.0. Both were checked in November 2026.

Where LiteLLM is stronger

It caches responses. Exact-match and semantic caching are free, with Redis, Valkey, Qdrant, S3 and other backends. agentgateway has no built-in response cache, so repetitive workloads like support bots pay full price for every request unless you add a cache yourself.

Its feature list is longer almost everywhere. LiteLLM supports more providers, more routing strategies (including cost-based and latency-based routing), more budget scopes and more observability integrations than agentgateway. If a feature exists in the category, LiteLLM probably has a setting for it.

Its limits hold across instances for free. Because LiteLLM keeps state in Postgres and Redis, budgets and rate limits apply across every replica without a paid tier. agentgateway's free controls stop at a single instance.

Its MCP permissions go one level deeper. LiteLLM can allow a tool but block specific parameters to it, and it layers permissions across six identity levels. agentgateway's rules work per tool.

It has the bigger community. More than 61,000 stars, weekly releases and well-known users mean most problems you hit have already been hit by someone else. You can also get a 30-day Enterprise trial key without talking to sales, which Solo doesn't offer.

Where agentgateway is stronger

It is much faster and smaller. On the same 1 vCPU test setup, agentgateway added about 2 to 5 ms per request, while LiteLLM added 12 to 15 ms at light load and started queueing well before 50 requests per second. agentgateway idles at about 18 MB of memory; LiteLLM used about 1.2 GB at 10 requests per second.

There is less to run. One binary, no database and no Redis is enough for a working gateway. LiteLLM's production setup is a load balancer, Postgres, Redis and a secret store, which is a normal cloud footprint but a lot more than a single container.

More security controls are free. JWT validation against your identity provider and PII masking are part of the open-source build. In LiteLLM, JWT authentication and per-team guardrails need an Enterprise license.

The project belongs to a foundation. agentgateway's code is Apache 2.0 under a Linux Foundation project, so it doesn't depend on one company's plans. LiteLLM is MIT-licensed but its maintainers are mostly BerriAI staff, and the enterprise folder is proprietary.

It fits Kubernetes natively. agentgateway implements the Gateway API with its own controller and CRDs, so routing config looks like any other cluster resource. LiteLLM ships Helm charts but no operator or CRDs.

Performance

We ran both gateways through the same harness: one container capped at 1 vCPU and 2 GB, a local mock provider with a fixed 50 ms response, and every figure measured against calling the mock directly. One caveat in LiteLLM's favour: its docs size production instances at 4 vCPU and 8 GB, so 1 vCPU is a tight fit for a Python proxy.

Both vendors publish their own numbers too. LiteLLM's benchmark page reports 1,170 requests per second at 8 ms p95 overhead across four 4 vCPU instances. Solo has posted head-to-head benchmarks against LiteLLM showing 0.44 ms against 30.6 ms at p99 at a fixed 3,000 requests per second, with 13 MiB of memory against 11.7 GiB. Those are vendor-run tests against mock backends, so read them as direction rather than proof. Our own numbers point the same way.

Does the gap matter? For a chat app making a few requests per second, probably not: a model call takes hundreds of milliseconds, and 10 ms of gateway time disappears in that. It matters when you run agents that make many calls in a loop, or when you're paying for the extra instances LiteLLM needs to handle the same load.

Pricing

Both are free to self-host. LiteLLM's open-source edition has no usage cap; you pay once you need SSO for more than five admin users, SCIM, audit logs, JWT authentication, per-team guardrails or the admin and worker split. LiteLLM Enterprise is an annual contract sized by gateway request volume, with no public price, and you can start a 30-day trial with an instant key.

agentgateway's free edition covers per-key budgets, PII masking, JWT validation and MCP authorization on a single instance. Solo Enterprise for agentgateway adds fleet-wide rate limits, a budget resource, a cost management UI, on-behalf-of identity, FIPS builds and support. It is also priced through sales, and there is no self-service trial. Our Solo AI gateway pricing breakdown has more detail.

The license is often the smaller cost for either one. With LiteLLM, budget for managed Postgres and Redis and the extra instances a Python proxy needs at scale. With agentgateway, budget for the enterprise edition once you need limits shared across instances.

Gotchas worth knowing

Pin LiteLLM versions and verify images. LiteLLM had two supply-chain compromises in 2026. On March 24, malicious PyPI releases 1.82.7 and 1.82.8 shipped a credential stealer. On October 21, a compromised v1.107.0 reached PyPI and Docker Hub for about two hours, and its images carried a valid signature. LiteLLM published full write-ups, rotated its signing key and changed its release process, but pinning exact versions is now basic hygiene.

Keep up with LiteLLM patches. A critical SQL injection (CVE-2026-42208) affected v1.81.16 through v1.83.6 and was fixed in v1.83.7. With weekly releases, plan to upgrade often. In November 2026 LiteLLM also started a long-term support pilot on the 1.108 line.

Decide who owns LiteLLM's config. With models stored in the database, changes made in the admin UI override the YAML file, so the running gateway can drift from what's in Git.

Redis is part of LiteLLM's correctness, not just its speed. Without it, each instance enforces its own rate limits rather than the limit you set.

agentgateway retries one more time than its docs suggest. A retry setting of 1 makes two provider calls, with no backoff unless you add one.

agentgateway streams can hang without a timeout. Set responseIdleTimeout (v1.6.0 and later) on streaming routes, or a provider that stalls mid-stream holds the connection open.

When to choose LiteLLM

• You want caching, cost-based routing or the widest provider list without paying for anything.
• You need budgets and rate limits that hold across many instances on the free tier.
• Your team is comfortable running Postgres and Redis and writing Python plugins.
• You want a large community and a self-service enterprise trial.

When to choose agentgateway

• Latency, memory or instance count matter, for example with agents making many calls.
• You want a gateway with no database to run, ideally on Kubernetes with the Gateway API.
• You need JWT validation and PII masking without an enterprise license.
• You prefer a foundation-governed project with an Apache 2.0 license.

The TrueFoundry alternative

Both gateways leave something for you to build. LiteLLM gives you features but hands you Postgres, Redis, patching and an enterprise license for SSO and audit logs. agentgateway is lean but makes you pay Solo for anything shared across instances, and it has no cache. The TrueFoundry AI Gateway runs as managed SaaS, as a gateway in your VPC with a control plane we host, or self-hosted in your VPC, on-premises or air-gapped. Gateway pods get configuration and access rules from the control plane in real time, along with centrally aggregated rate-limit and budget counters, so limits apply across replicas without a paid upgrade.

It exposes 1,000+ LLMs through one OpenAI-compatible API, routes to self-hosted models from the same control plane, and governs LLM, MCP and agent traffic with tool-level access control and human approval before destructive tool calls. Budgets can be scoped by team, model or provider account, and SSO, RBAC and audit logging are built in. If you're comparing open-source gateways because you need both cost control and agent governance, see how the TrueFoundry AI Gateway handles both or book a demo.

Related reading

• agentgateway (formerly Gloo AI Gateway): an honest review
• LiteLLM alternatives
• Solo AI gateway pricing: a complete breakdown
• Solo.io competitors and alternatives
• Best MCP gateways
• What is an LLM gateway?

Conclusion

LiteLLM vs agentgateway comes down to what you value more: breadth or efficiency. LiteLLM gives you the most features for free, including caching and shared limits, at the cost of a heavier stack and a busy patching schedule. agentgateway gives you a fast, small gateway with strong agent and MCP controls, but anything shared across instances is a paid feature. If you want both without running either yourself, book a TrueFoundry demo.