Chat Completions: Caching & Reasoning

Prompt Caching

Prompt caching reduces processing time and costs by reusing previously computed prefixes. When you send the same or similar prompts repeatedly (e.g. a large system prompt, a shared context document, or a set of tool definitions), the provider can skip reprocessing the cached portion and only compute the new tokens. This leads to lower latency and reduced costs on subsequent requests.

You can send cache_control for any model. The AI Gateway handles the per provider caching logic internally, forwarding it to providers that read cache_control and stripping it for providers that cache prefixes on their own. This means you can write one request and use it across providers without worrying about compatibility.

How to Enable

Add "cache_control": {"type": "ephemeral"} to any system prompt, user message content block, or tool definition:

response = client.chat.completions.create(
    model="anthropic/claude-opus-4-6",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "Your large system prompt here...",
                    "cache_control": {"type": "ephemeral"},
                },
            ],
        },
        {"role": "user", "content": "Your question here"},
    ]
)

response = client.chat.completions.create(
    model="anthropic/claude-opus-4-6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Your large context or reference text here...",
                    "cache_control": {"type": "ephemeral"},
                },
                {
                    "type": "text",
                    "text": "Now answer my question about the above.",
                },
            ],
        },
    ]
)

response = client.chat.completions.create(
    model="anthropic/claude-opus-4-6",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get current weather",
                "parameters": {
                    "type": "object",
                    "properties": {"location": {"type": "string"}},
                    "required": ["location"]
                }
            },
            "cache_control": {"type": "ephemeral"}
        }
    ]
)

Enabling Caching with a Header

If you can’t edit the request body — for example when the payload is built by a framework or an off-the-shelf agent — send the x-tfy-cache-control header instead. The AI Gateway places a single cache breakpoint at the end of the prompt, so the entire prefix (tool definitions, system prompt, and earlier messages) is cached.

curl {GATEWAY_BASE_URL}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-truefoundry-api-key" \
  -H "x-tfy-cache-control: {\"type\": \"ephemeral\"}" \
  -d '{
    "model": "anthropic-main/claude-opus-4-6",
    "messages": [
      {"role": "system", "content": "<LONG_SYSTEM_PROMPT_OVER_THE_MIN_TOKENS>"},
      {"role": "user", "content": "Summarize the key rules from the system prompt."}
    ]
  }'

response = client.chat.completions.create(
    model="anthropic-main/claude-opus-4-6",
    messages=[
        {"role": "system", "content": "<LONG_SYSTEM_PROMPT_OVER_THE_MIN_TOKENS>"},
        {"role": "user", "content": "Summarize the key rules from the system prompt."},
    ],
    extra_headers={
        "x-tfy-cache-control": '{"type": "ephemeral", "ttl": "1h"}',
    },
)

The header value is a JSON object with the same shape as the cache_control field you would have written in the body:

Field	Required	Description
`type`	Yes	Cache type forwarded to the provider. Claude models accept `ephemeral`.
`ttl`	No	Cache duration in Anthropic’s format — `5m`, `1h`, `2d`. Ignored by providers that don’t accept a duration (including Bedrock chat completions, which use `cachePoint`).

The header is a hint, and the AI Gateway only acts on it where it changes anything:

On Claude models from Anthropic, Claude Platform on AWS, Azure AI Foundry, AWS Bedrock, Google Vertex, and Databricks, the breakpoint is injected in the format that provider reads.
On OpenAI, Gemini, Groq, xAI and other providers that cache repeated prefixes on their own, nothing is injected and the request is forwarded unchanged — see Provider Support below.
A cache_control in the body always wins. If the request already marks any block, the header is ignored, so hand-placed breakpoints are never overwritten.

The value must be valid JSON and, if you set ttl, it must match the 5m / 1h / 2d format — otherwise the AI Gateway rejects the request with a 400.

x-tfy-cache-control is not the same as x-tfy-cache-config, which turns on the AI Gateway’s own exact-match and semantic response cache. They work at different layers and can be used together.

Provider Support

There are three caching styles. The first two are marker-based — you send cache_control and the provider caches based on it (this uses the same explicit vs automatic distinction as the Messages API):

Explicit — you place cache_control on individual blocks (a system block, a message content block, or a tool definition).
Automatic — you place a single cache_control field at the top level of the request, or send the x-tfy-cache-control header, and the AI Gateway puts the breakpoint at the end of the prompt for you.
Provider-managed — the provider caches repeated prefixes on its own with no markup, so the AI Gateway strips any cache_control you send.

Provider	Caching style	How `cache_control` is handled
Anthropic (direct), Azure AI Foundry (Claude)	Explicit + automatic	Forwarded to the provider unchanged as native Anthropic `cache_control` (optional `ttl` supported).
Google Vertex (Claude), Databricks (Claude)	Explicit + automatic	Forwarded to the provider unchanged as native Anthropic `cache_control` (optional `ttl` supported).
AWS Bedrock (Claude)	Explicit + automatic	Chat completions run through Bedrock’s Converse API, so each block-level `cache_control` is translated into a native `cachePoint` marker.
OpenAI / Azure OpenAI	Provider-managed	`cache_control` stripped before forwarding. Optionally pass `prompt_cache_key`.
Google Gemini / Google Vertex (Gemini)	Provider-managed	`cache_control` stripped before forwarding.
Groq, xAI, and others	Provider-managed	`cache_control` stripped before forwarding.

The examples above use explicit caching (block-level cache_control), which works on every Claude provider. On chat completions, a top-level cache_control is never sent to the provider as-is — the AI Gateway relocates it onto the last cacheable block, which is the one form every marker-based provider honours. The x-tfy-cache-control header behaves the same way. See Messages API caching for how the same request looks on /messages.

Anthropic (and Bedrock / Vertex / Azure / Databricks Claude)

For Anthropic (direct), Google Vertex (Claude), Azure AI Foundry (Claude), and Databricks (Claude), cache_control is forwarded to the provider unchanged. You can also include an optional ttl field (e.g. "5m" or "1h") to control cache duration.For AWS Bedrock, the AI Gateway translates each cache_control block into Bedrock’s native Converse cachePoint format, so your request code stays the same. Converse cachePoint markers do not carry a duration, so any ttl you set is ignored on Bedrock chat completions.

On the Messages API (/messages), Bedrock Claude models are served through the InvokeModel API with the native Anthropic body instead of Converse. There, cache_control (and ttl) is forwarded unchanged rather than translated to cachePoint.

Anthropic enforces a minimum content length for caching to take effect; shorter prompts accept the cache_control hint but are not cached:

Minimum tokens	Models
`4096`	Claude Mythos Preview, Opus 4.7, Opus 4.6, Opus 4.5, Haiku 4.5
`2048`	Sonnet 4.6, Haiku 3.5, Haiku 3
`1024`	Sonnet 4.5, Opus 4.1, Opus 4, Sonnet 4, Sonnet 3.7

For Amazon Titan and Nova models on Bedrock, cache_control on tool definitions is automatically skipped since these models do not support cache points on tools.

OpenAI / Azure OpenAI

OpenAI and Azure OpenAI are provider-managed — they cache matching prefixes on their own. No cache_control markup is needed, and the AI Gateway strips it before forwarding.You can optionally pass prompt_cache_key to group requests that share a common prefix, improving cache hit rates:

response = client.chat.completions.create(
    model="openai-main/gpt-4o",
    messages=[{"role": "user", "content": "Your prompt here"}],
    prompt_cache_key="optional-custom-key"
)

prompt_cache_key is only supported for OpenAI and Azure OpenAI.

Gemini / Groq / xAI

These providers are provider-managed — they handle caching on their own end. No cache_control markup is needed, and the AI Gateway strips it before forwarding.Cached token counts are still reported in the response usage when the provider returns them.

Cache Usage in Responses

When caching is active, the response usage object includes cached token counts:

{
  "usage": {
    "prompt_tokens": 1500,
    "completion_tokens": 200,
    "total_tokens": 1700,
    "prompt_tokens_details": {
      "cached_tokens": 1200
    },
    "cache_read_input_tokens": 1200,
    "cache_creation_input_tokens": 300
  }
}

prompt_tokens_details.cached_tokens: tokens served from cache. Available across all providers that report cache usage.
cache_read_input_tokens / cache_creation_input_tokens: Anthropic style fields, present for Anthropic, Bedrock, and Groq when values are non zero.

Reasoning Models

TrueFoundry AI Gateway provides access to model reasoning processes through thinking/reasoning tokens, available for models from multiple providers including Anthropic,OpenAI,Azure OpenAI,Groq, xAI and Vertex. These models expose their internal reasoning process, allowing you to see how they arrive at conclusions. The thinking/reasoning tokens provide step-by-step insights into the model’s cognitive process.

Supported Reasoning Models

OpenAI

Supported models: o4-mini, o4-preview, o3 model family, o1 model family, gpt-5-mini, gpt-5-nano, gpt-5

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="openai-main/o4-mini",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "medium", "low"
    max_tokens=8000
)

print(response.choices[0].message.content)

To reach the model’s reasoning, the AI Gateway serves these requests through OpenAI’s Responses API upstream while keeping your request and response on the Chat Completions contract. See Reasoning on OpenAI and Azure OpenAI.

Azure OpenAI

Supported models: gpt-5, gpt-5-mini, gpt-5-nano, o3-pro, codex-mini, o4-mini, o3, o3-mini, o1, o1-mini

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="azure-openai-main/o3-mini",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "medium", "low"
    max_tokens=8000
)

print(response.choices[0].message.content)

Azure deployments need an api-version of 2025-03-01-preview or newer to return reasoning. See Reasoning on OpenAI and Azure OpenAI.

Anthropic

Supported models: Claude Opus 4.1 (claude-opus-4-1-20250805), Claude Opus 4 (claude-opus-4-20250514), Claude Sonnet 4 (claude-sonnet-4-20250514), Claude Sonnet 3.7 (claude-3-7-sonnet-20250219)
via Anthropic, AWS Bedrock, and Google Vertex AI

Using OpenAI SDK

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="anthropic-main/claude-3-7-sonnet",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "medium", "low", "none"
    max_tokens=8000
)

print(response.choices[0].message.content)

For Anthropic models (from Anthropic, Google Vertex AI, AWS Bedrock), TrueFoundry automatically translates the reasoning_effort parameter into Anthropic’s native thinking parameter format since Anthropic doesn’t support the reasoning_effort parameter directly.The translation uses the max_tokens parameter with the following ratios:

none: 0% of max_tokens
low: 30% of max_tokens
medium: 60% of max_tokens
high: 90% of max_tokens

Using Direct API Calls with Native `thinking` Parameter

For more precise control with Anthropic models, you can use the native thinking parameter directly:

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="anthropic-main/claude-3-7-sonnet",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    max_tokens=8000,
    extra_body={
        "thinking": {
            "type": "enabled",
            "budget_tokens": 1000
        }
    }
)

print(response.choices[0].message.content)

Groq

Supported models: OpenAI GPT-OSS 20B (openai/gpt-oss-20b), OpenAI GPT-OSS 120B (openai/gpt-oss-120b), Qwen 3 32B (qwen/qwen3-32b), DeepSeek R1 Distil Llama 70B (deepseek-r1-distill-llama-70b)

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="groq-main/deepseek-r1-distill-llama-70b",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "medium", "low"
    max_tokens=8000
)

print(response.choices[0].message.content)

xAI

Supported models: grok-3-mini (with reasoning_effort parameter), grok-4-0709, grok-4-1-fast-reasoning, grok-4-fast-reasoning (reasoning built-in)For grok-3-mini, you can use the reasoning_effort parameter to control reasoning depth. Other Grok models like grok-4-0709 have reasoning capabilities built-in but do not support the reasoning_effort parameter.

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

# For grok-3-mini with reasoning_effort parameter
response = client.chat.completions.create(
    model="xai-main/grok-3-mini",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "low" (only for grok-3-mini)
    max_tokens=8000
)

print(response.choices[0].message.content)

The reasoning_effort parameter is only supported for grok-3-mini. For other Grok models like grok-4-0709 and grok-4-1-fast-reasoning, reasoning is built-in and the reasoning_effort parameter should not be used. Reasoning tokens are included in the usage metrics for all reasoning-capable models.Parameter Restrictions: Reasoning models (like grok-4-0709 and grok-4-1-fast-reasoning) do not support presence_penalty, frequency_penalty, or stop parameters. Using these parameters with reasoning models will result in an error.

Gemini

Supported models: All Gemini 2.5 Series Models.These models can be accessed from Google Vertex or Google Gemini Providers

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="vertex-main/gemini-2-5-pro",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    reasoning_effort="high",  # Options: "high", "medium", "low", "none"
    max_tokens=8000
)

print(response.choices[0].message.content)

For Gemini models (from Anthropic, Google Vertex AI, AWS Bedrock), TrueFoundry automatically translates the reasoning_effort parameter into Gemini’s native thinking parameter format since Gemini doesn’t support the reasoning_effort parameter directly.The translation uses the max_tokens parameter with the following ratios:

none: 0% of max_tokens
low: 30% of max_tokens
medium: 60% of max_tokens
high: 90% of max_tokens

Note: Gemini 2.5 Pro and 2.5 Flash comes with reasoning on by default.

Using Direct API Calls with Native `thinking` Parameter

For more precise control with Gemini models, you can use the native thinking parameter directly:

from openai import OpenAI

client = OpenAI(
    api_key="TFY_API_KEY",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="vertex-main/gemini-2-5-pro",
    messages=[
        {"role": "user", "content": "How to compute 3^3^3?"}
    ],
    max_tokens=8000,
    extra_body={
        "thinking": {
            "type": "enabled",
            "budget_tokens": 1000
        }
    }
)

print(response.choices[0].message.content)

Reasoning on OpenAI and Azure OpenAI

OpenAI exposes a model’s reasoning only through its Responses API — a plain Chat Completions call to gpt-5.x cannot return it. So when a request needs reasoning, the AI Gateway serves it through the provider’s Responses API upstream and converts the reply back. Your code does not change. You keep sending Chat Completions requests and reading Chat Completions responses; the switch happens inside the AI Gateway.

When it applies

The model must support the Responses API. Beyond that, any one of these turns it on:

Trigger	Example
The request asks for reasoning	`reasoning_effort` set to anything other than `none`, or a `thinking` field
You opt in with a header	`x-tfy-openai-use-responses-api: true`
The model only speaks Responses	`codex-mini`, `gpt-5.5-pro`

That last row is worth calling out: responses-only models used to be rejected on /chat/completions. They now work, so you can reach them through the same OpenAI SDK client as every other model. Use the header when you want reasoning behaviour without changing the body — for example when the payload is built by a framework or an off-the-shelf agent:

curl {GATEWAY_BASE_URL}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-truefoundry-api-key" \
  -H "x-tfy-openai-use-responses-api: true" \
  -d '{
    "model": "openai-main/gpt-5",
    "messages": [{"role": "user", "content": "How to compute 3^3^3?"}]
  }'

response = client.chat.completions.create(
    model="openai-main/gpt-5",
    messages=[{"role": "user", "content": "How to compute 3^3^3?"}],
    extra_headers={"x-tfy-openai-use-responses-api": "true"},
)

What comes back

Field	Description
`message.reasoning_content`	Plain-text summary of the model’s reasoning
`message.thinking_blocks`	Array of `{type: "reasoning", id, encrypted_content}` objects — replay these to continue a reasoning chain
`usage.completion_tokens_details.reasoning_tokens`	Tokens the model spent thinking, billed as output tokens

{
  "message": {
    "role": "assistant",
    "content": "3^3^3 = 7,625,597,484,987",
    "reasoning_content": "Exponentiation is right-associative: 3^3 = 27, then 3^27...",
    "thinking_blocks": [
      {
        "type": "reasoning",
        "id": "rs_68b1f...",
        "encrypted_content": "gAAAAAB..."
      }
    ]
  }
}

OpenAI returns its reasoning encrypted rather than in plain text, so thinking_blocks here carry id and encrypted_content instead of the thinking and signature fields you get from Claude and Gemini. Both shapes are replayed the same way — see Multi-Turn Conversations. When streaming, reasoning_content arrives as deltas before the content, and the thinking_blocks entry is emitted as its own chunk once the reasoning item is complete.

Multi-turn conversations

Echo the assistant message back unchanged on the next turn. Without it the model starts each turn from scratch, which matters most in agent loops where the model builds on its earlier thinking.

response = client.chat.completions.create(
    model="openai-main/gpt-5",
    messages=[{"role": "user", "content": "What is 3^3^3?"}],
    reasoning_effort="high",
)

# Captures content, tool_calls and thinking_blocks in one step
assistant_message = response.choices[0].message.model_dump(exclude_none=True)

response2 = client.chat.completions.create(
    model="openai-main/gpt-5",
    messages=[
        {"role": "user", "content": "What is 3^3^3?"},
        assistant_message,
        {"role": "user", "content": "Now explain why exponentiation is right-associative."},
    ],
    reasoning_effort="high",
)

A thinking block is only replayable if it still has both its id and encrypted_content. Blocks missing either are skipped, so serialize the assistant message as returned rather than rebuilding it by hand.

Parameters that are dropped

The Responses API has no equivalent for some Chat Completions parameters. When a request is served this way, these are dropped and the request still goes through — reasoning is never silently disabled to preserve them: n (above 1), best_of, frequency_penalty, presence_penalty, logit_bias, logprobs, top_logprobs, stop, seed, audio, modalities (non-text), prediction

If your request depends on one of these — sampling with seed, cutting generation with stop, or scoring with logprobs — leave reasoning_effort unset and don’t send the header. The request then stays on Chat Completions and the parameters are honoured.

Inert values are ignored rather than dropped, so n: 1, logprobs: false, and zeroed penalties never change anything.

Azure OpenAI requirements

Two extra conditions apply on Azure:

The deployment’s api-version must be 2025-03-01-preview or newer. Older versions reject /responses outright, so the AI Gateway keeps those requests on Chat Completions.
The deployment’s foundation model must be listed as Responses-capable in the model catalogue. Azure deployment names are arbitrary, so the AI Gateway matches on the foundation model rather than the deployment name.

Response Format

When reasoning tokens are enabled, the response includes both thinking and content sections:

{
  "id": "1742890579083",
  "object": "chat.completion",
  "created": 1742890579,
  "model": "",
  "provider": "aws",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Therefore: 3^3^3 = 7,625,597,484,987",
        "reasoning_content": "Exponentiation is right-associative: compute 3^3 = 27, then 3^27..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 45,
    "completion_tokens": 180,
    "total_tokens": 225
  }
}

Streaming with Reasoning Tokens

For streaming responses, the thinking section is always sent before the content section.

Get Started

LLM Gateway

MCP Registry and Gateway

Skills Registry

Prompt Registry

Guardrails and Security

Observability

Deployment

Admin Guide

Chat

Messages

Embeddings

Rerank

Responses

Image

Audio

Batch

Files

Fine-tuning

Moderations

Models

Chat Completions: Caching & Reasoning

Prompt Caching

How to Enable

Enabling Caching with a Header

Provider Support

Cache Usage in Responses

Reasoning Models

Supported Reasoning Models

Using OpenAI SDK

Using Direct API Calls with Native `thinking` Parameter

Using Direct API Calls with Native `thinking` Parameter

Reasoning on OpenAI and Azure OpenAI

When it applies

What comes back

Multi-turn conversations

Parameters that are dropped

Azure OpenAI requirements

Response Format

Streaming with Reasoning Tokens

​Prompt Caching

​How to Enable

​Enabling Caching with a Header

​Provider Support

​Cache Usage in Responses

​Reasoning Models

​Supported Reasoning Models

​Using OpenAI SDK

​Using Direct API Calls with Native thinking Parameter

​Using Direct API Calls with Native thinking Parameter

​Reasoning on OpenAI and Azure OpenAI

​When it applies

​What comes back

​Multi-turn conversations

​Parameters that are dropped

​Azure OpenAI requirements

​Response Format

​Streaming with Reasoning Tokens

Prompt Caching

How to Enable

Enabling Caching with a Header

Provider Support

Cache Usage in Responses

Reasoning Models

Supported Reasoning Models

Using OpenAI SDK

Using Direct API Calls with Native `thinking` Parameter

Using Direct API Calls with Native `thinking` Parameter

Reasoning on OpenAI and Azure OpenAI

When it applies

What comes back

Multi-turn conversations

Parameters that are dropped

Azure OpenAI requirements

Response Format

Streaming with Reasoning Tokens