What Is an OpenAI-Compatible API? The Interface That Unifies LLM Providers

Pick almost any LLM product shipped in the last three years and inspect its API. The odds are high that it accepts a request with a model field, a messages array of {role, content} objects, and temperature, and returns a choices array with a usage block counting tokens. That shape did not come from a standards body. It came from OpenAI's Chat Completions API, and it won.

For a platform team, "OpenAI-compatible" is a load-bearing phrase. It is the reason you can evaluate a new open-weight model on Friday and have it serving traffic Monday without rewriting your application. It is also a phrase that quietly overpromises: two endpoints can both be "OpenAI-compatible" and still disagree on how tool calls are returned, how tokens are counted, and whether the parameter you set actually does anything.

This post explains what the standard actually is, what "compatible" guarantees and what it does not, and how a gateway turns many partially-compatible providers into one endpoint you can build against with confidence.

Want one OpenAI-compatible endpoint across every provider, with failover and cost tracking? Book a demo.

How the Chat Completions schema became the default

Standards usually win for one of two reasons: they are mandated, or they arrive first and get adopted before anyone designs an alternative. The Chat Completions schema is firmly the second kind.

OpenAI published the /v1/chat/completions endpoint and, crucially, shipped official Python and JavaScript SDKs that wrapped it. Those SDKs got embedded into tutorials, starter templates, framework integrations, and production codebases across the industry. By the time competing providers had models worth serving, the integration surface was already standardized in practice: millions of lines of code already knew how to call client.chat.completions.create(...).

A new provider then faced a choice. Define your own elegant API and ask the entire ecosystem to learn it, or implement OpenAI's schema and inherit every SDK, framework, and tool for free. Nearly everyone chose the second path. Open-weight serving stacks adopted it so that anything running on them was instantly reachable from existing code. The result is a de-facto standard with no standards document: the schema is "whatever the OpenAI SDK sends and expects back."

This is why the phrase matters more than it looks. Compatibility is not a feature a vendor adds for polish. It is the entry ticket to an ecosystem, and it shapes how you can architect around model choice.

What "OpenAI-compatible" actually means

Strip it down and compatibility is a contract about four things: the endpoint, the request shape, the response shape, and streaming.

The endpoint. A compatible service accepts POST /v1/chat/completions with an Authorization: Bearer <key> header. Many also implement /v1/models, /v1/embeddings, and /v1/completions. The base URL changes per provider; the path does not.

The request shape. At minimum: a model string and a messages array of objects with a role (system, user, assistant, tool) and content. Optional parameters the client can send include temperature, top_p, max_tokens (newer OpenAI models use max_completion_tokens), stream, stop, tools, tool_choice, and response_format.

The response shape. A JSON object with a choices array; each choice has an index, a message with role and content (and tool_calls when the model calls a function), and a finish_reason (stop, length, tool_calls, content_filter). A usage object reports prompt_tokens, completion_tokens, and total_tokens.

Streaming. With stream: true, the response is a text/event-stream of server-sent events. Each data: line carries a chunk whose choices[].delta holds the incremental token(s). The stream terminates with a literal data: [DONE].

A worked example: same body, different model

Here is the concrete payoff. This request body targets OpenAI:

POST https://api.openai.com/v1/chat/completions
Authorization: Bearer sk-...

{
  "model": "gpt-4.1",
  "messages": [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user", "content": "Summarize MCP in one sentence."}
  ],
  "temperature": 0.2,
  "stream": false
}

To run the same prompt against a different provider through a compatible endpoint, you change two things: the base URL and the model value. The messages, temperature, and stream fields are untouched. Your parsing code, which reads choices[0].message.content and usage.total_tokens, does not change at all. That is the entire value proposition in one diff: model choice becomes a string, not a rewrite.

The response, whichever backend served it, comes back as:

{
  "id": "chatcmpl-...",
  "choices": [
    {"index": 0,
     "message": {"role": "assistant", "content": "MCP is an open protocol..."},
     "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 24, "completion_tokens": 18, "total_tokens": 42}
}

The gap between "compatible" and "identical"

Here is where teams get burned. "OpenAI-compatible" is a claim about the schema, not a guarantee of identical behavior. The envelope matches; the contents can differ in ways that only surface under load or in edge cases.

Tool-calling dialects. The biggest divergence. OpenAI returns function calls in message.tool_calls, each with an id, a type, and a function object whose arguments is a stringified JSON blob you must parse. Some compatible backends return arguments as an already-parsed object, emit the older single function_call field, stream tool-call deltas differently, or handle parallel tool calls inconsistently. Code that blindly JSON.parsees the arguments will throw on a backend that already parsed them.

Token accounting.usage counts are computed by each provider's own tokenizer. The same prompt sent to two models yields different prompt_tokens because the tokenizers differ. Worse, some backends omit usage entirely on streaming responses unless you set a flag, so cost tracking that relies on it silently reads zero.

Silent no-op parameters. You can send seed, logit_bias, frequency_penalty, presence_penalty, or response_format: {type: "json_schema"} to a compatible endpoint and get a 200 OK back, with the parameter completely ignored. The endpoint accepted the field because the schema allows it; the backend just does not implement it. This is the nastiest class of bug because nothing errors. Your "deterministic" pipeline that relies on seed is quietly nondeterministic.

System-prompt and role handling. Some backends fold the system message into the first user turn, or support only one system message, or treat an empty content differently. Multi-turn behavior can drift.

Finish-reason semantics. When a model stops because it wants to call a tool, OpenAI reports finish_reason: "tool_calls". Not every backend uses the same string, which breaks agent loops that branch on it.

The table below separates what the standard reliably gives you from what you must verify per provider.

The takeaway is not "compatibility is fake." It is "compatibility gets you 90% for free and you must test the last 10% per provider." That last 10% is precisely the work a gateway can absorb so your application never sees it.

See how TrueFoundry lets you swap models by name without touching application code. Book a demo.

One endpoint across every provider: the gateway pattern

If every provider is mostly compatible but subtly different, the clean architecture is to stop making your application talk to each provider. Put one OpenAI-compatible endpoint in front of all of them and let that layer handle translation and normalization.

This is what an AI gateway does, and it is structurally the same idea as an LLM proxy: a single logical front door, many backends behind it. TrueFoundry exposes a unified OpenAI-compatible API in front of every provider. Your application keeps sending the Chat Completions shape it already knows. To switch models, you change the model value to a logical name the gateway resolves; everything else, request body, SDK, parsing, stays constant.

Because the gateway normalizes the differences described above, your code stops caring whether a given backend stringifies tool-call arguments or reports usage on streams. The gateway presents one consistent dialect. Open-source libraries like LiteLLM popularized this translation layer in code; our explainer on what LiteLLM is covers that approach and where a managed gateway goes further.

But the reason to adopt the pattern is not just a tidy interface. Once every request flows through one endpoint, you unlock capabilities that are impossible to add to scattered per-provider integrations:

Routing. A logical model name can be backed by several real targets, and the gateway picks among them by latency, weight, or priority. See what an LLM router is for cost- and quality-aware model selection.

Fallback. If a provider returns a 500 or times out, the gateway retries in-request against a fallback target, so a single provider outage does not surface as an error to your user. That is the subject of LLM fallback.

Caching. Exact and semantic caching at the gateway cut both latency and spend for repeated or near-duplicate prompts, transparently, behind the same endpoint.

Observability and cost attribution. Because every request passes through one place, you get cost, token, and latency metrics per request, tagged by user, team, and model, feeding LLM observability and cost attribution to team budgets. A direct per-provider integration gives you none of this without bespoke instrumentation on each one.

None of these required you to adopt a new API. They came from standardizing on the one interface you already use. If you are weighing options, how to choose an AI gateway lays out the criteria.

Direct-to-provider versus a compatible gateway

The decision is not religious; it is about how many providers you touch and how much you care about resilience and cost visibility.

Direct integration is defensible when you call exactly one provider and never will call another. The moment you want a second model, resilience against an outage, or a single source of truth for cost, the gateway pays for itself.

Where teams get this wrong

Assuming compatible means identical. The schema matches; the behavior does not always. Teams port a prompt to a cheaper backend, see it "work," and ship, only to discover weeks later that seed was ignored and outputs were never reproducible. Test the parameters you actually depend on against each backend.

Trustingusagefor cost without checking streaming. Several backends omit the usage block on streamed responses unless explicitly asked. Cost dashboards built on it read zero and look great until the real bill arrives. Verify that usage is present on the exact request style you run in production.

Hardcoding the base URL and SDK per provider. This recreates the sprawl the standard was supposed to eliminate. Every new model becomes an integration project and every provider outage becomes your outage. Standardize on one endpoint and make model choice configuration, not code.

Parsing tool-call arguments without guarding the format. Some backends return arguments as a string, others as an object. Blind JSON.parse throws on the latter. Normalize the shape before you parse, or let a gateway normalize it for you.

Treating the gateway as pure plumbing. The endpoint is the easy part. The value is what one chokepoint enables, routing, fallback, caching, budgets, and attribution, which you cannot retrofit cheaply onto N direct integrations.

Conclusion

The OpenAI Chat Completions schema became the standard not by decree but by arriving first and getting adopted everywhere. "OpenAI-compatible" is the ticket into that ecosystem: send the familiar request shape, parse the familiar response. It buys you enormous leverage, model choice as a configuration string, as long as you respect that compatible is not identical and test the behaviors you actually depend on.

The cleanest way to get the leverage without the sharp edges is to standardize on one OpenAI-compatible endpoint in front of every provider. Your application keeps the interface it already knows, and the gateway absorbs the per-provider quirks while adding routing, fallback, caching, budgets, and cost attribution you could never bolt onto scattered integrations. One interface in; every model behind it.

Unify every LLM provider behind one OpenAI-compatible API. Book a demo.