> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Native SDK Support

> Use OpenAI, AzureOpenAI, Google Gen AI, Anthropic, and boto3 SDKs with the TrueFoundry AI Gateway.

You can use these native provider SDKs with the TrueFoundry AI Gateway to send requests to your configured models. **Tracing**, **cost tracking**, and **rate budget limits** are supported for all of them.

| SDK | Models supported | Cost tracking | Rate/Budget limits | Routing configuration |
| - | - | - | - | - |
| [OpenAI](#openai-sdk) | All models | Yes | Yes | Yes |
| [Google Gen AI](#google-gen-ai-sdk) | Gemini, Vertex AI models | Yes | Yes | No |
| [Legacy Vertex AI](#legacy-vertex-ai-sdk) | Vertex AI Gemini models | Yes | Yes | Yes |
| [Anthropic](#anthropic-sdk) | Anthropic models only | Yes | Yes | Yes |
| [AzureOpenAI](#azureopenai-sdk) | Azure OpenAI models only | Yes | Yes | Yes |
| [boto3 (Bedrock Runtime)](#boto3-sdk-aws-bedrock-runtime) | Bedrock models | Yes | Yes | No |
| [langchain\_aws](#langchain-aws-sdk) | Bedrock models | Yes | Yes | No |

<Info>
  **Copy these examples from this page.** For chat models, the Playground's **Code Snippet** panel only generates OpenAI-compatible snippets — OpenAI Library, LangChain, LangGraph, LlamaIndex, LiteLLM, curl, NodeJS, Go, Rust, and similar. It does not generate the Google Gen AI, Anthropic, boto3, or `langchain_aws` setups shown below.

  The panel also has no snippet for the [Responses API](/docs/ai-gateway/responses-api) (`/responses`) or for `/completions`; use the examples in those guides instead.

  Realtime and audio models are the exception: the Playground does generate native snippets for those, including Google Gen AI, OpenAI Realtime, Azure Voice Live, ElevenLabs, Deepgram, and Cartesia.
</Info>

## OpenAI SDK

Use the standard OpenAI client with the AI Gateway base URL and your TrueFoundry API key. See [Chat Completions - Getting Started](/docs/ai-gateway/chat-completions-overview#getting-started) for the full example:

```python lines theme={"dark"}
from openai import OpenAI

client = OpenAI(
    api_key="your_truefoundry_api_key",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="openai-main/gpt-4o-mini", # tfy model name
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)

print(response.choices[0].message.content)
```

## Google Gen AI SDK

The [Google Gen AI SDK](https://github.com/googleapis/python-genai) can be used with **Gemini** and **Vertex AI** via the AI Gateway by setting a custom base URL.

<Note>
  **Model ID must match the model name:** When using the Google Gen AI SDK with
  the AI Gateway, the model's display name in TrueFoundry must match the model
  ID, because the AI Gateway authorizes your API key against the model. You only
  need to pass the **actual model ID** (e.g. `gemini-2.5-flash`) in your
  request.
</Note>

<Note>
  **Routing configuration** is not supported when using the Google Gen AI SDK
  with the AI Gateway.
</Note>

```python lines theme={"dark"}
from google import genai
from google.genai import types

client = genai.Client(
    api_key="TFY_API_KEY",
    http_options=types.HttpOptions(
        base_url="{GATEWAY_BASE_URL}/gemini/{providerAccountName}/proxy"
    )
)

response = client.models.generate_content(
    model="gemini-2.5-flash",  # actual model id
    contents="How does AI work?",
)

print(response.text)
```

Replace `{providerAccountName}` with your **Gemini** or **Vertex AI** model account name from the AI Gateway.

## Legacy Vertex AI SDK

The legacy [`vertexai.generative_models`](https://cloud.google.com/python/docs/reference/vertexai/latest)
Python SDK can call Vertex AI models through the AI Gateway. Pin a pre-removal
`google-cloud-aiplatform` version when using it. New integrations should use
[`google-genai`](https://pypi.org/project/google-genai/) instead: Google deprecated the legacy
generative AI modules and announced their removal for June 24, 2026.

<Note>
  **Required gateway headers:** pass the TFY model FQN through
  `x-tfy-model-name` and the TFY credential through `x-goog-api-key` in
  `request_metadata`. The SDK's `model` value remains the native Vertex model
  ID.
</Note>

<Warning>
  The gateway's configured provider account supplies the upstream Google
  credentials, so this example intentionally uses `AnonymousCredentials()`.
</Warning>

```python lines theme={"dark"}
import vertexai
from google.auth.credentials import AnonymousCredentials
from vertexai.generative_models import GenerativeModel

tfy_model_name = "tfy-ai-vertex/google-gemini-3.5-flash-lite"
tfy_api_key = "your_truefoundry_api_key"

vertexai.init(
    project="your-gcp-project",
    location="us-central1",
    # The gateway authenticates upstream with its configured provider account.
    credentials=AnonymousCredentials(),
    # No https://. The SDK adds it.
    api_endpoint="{GATEWAY_HOST}/api/llm/vertex-ai",
    api_transport="rest",
    request_metadata=[
        ("x-goog-api-key", tfy_api_key),
        ("x-tfy-model-name", tfy_model_name),
    ],
)

model = GenerativeModel("google-gemini-3.5-flash-lite")

response = model.generate_content("Hello from Vertex AI.")

print(response.text)
```

The gateway uses the provider account and model from `x-tfy-model-name` for routing, credentials,
and upstream project/region configuration. The `project` and `location` passed to the SDK are
required to construct its native request path, but do not override the gateway's configured
provider account. Install the legacy client with a pinned pre-removal version:
`python3 -m pip install "google-cloud-aiplatform<2"`.

The SDK appends this native Vertex path below `api_endpoint`:

```text theme={"dark"}
/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent
```

The gateway replaces the SDK-supplied project, location, and native model path
with the configured provider-account values selected by `x-tfy-model-name`.

Supported methods include:

* `GenerativeModel.generate_content`
* `GenerativeModel.generate_content(..., stream=True)`
* `GenerativeModel.start_chat().send_message`
* Function calling and multimodal content
* `GenerativeModel.count_tokens`

For this gateway proxy mode, `AnonymousCredentials()` avoids a local Google credential lookup:
the gateway's configured Vertex provider account authenticates the upstream request. If you call
Google directly instead of through the gateway, use ADC, `GOOGLE_APPLICATION_CREDENTIALS`, or
explicit service-account credentials.

## Anthropic SDK

Use the Anthropic client with the AI Gateway base URL and your TrueFoundry API key to call Claude models:

```python lines theme={"dark"}
from anthropic import Anthropic

BASE_URL = "{GATEWAY_BASE_URL}"
API_KEY = "your-truefoundry-api-key"

client = Anthropic(
    api_key=API_KEY,
    base_url=BASE_URL,
    default_headers={
        "Authorization": f"Bearer {API_KEY}"
    }
)

def generate_response():
    response = client.messages.create(
        model="anthropic/claude-3-5",  # tfy model name
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Hello, Claude! Please explain quantum computing in simple terms."
            }
        ]
    )

    print(response.content)

generate_response()
```

## AzureOpenAI SDK

Use the Python `AzureOpenAI` client (from the `openai` package) with the AI Gateway to call **Azure OpenAI** models with the legacy, deployment-scoped Azure surface.

<Note>
  **Authentication:** Pass your TrueFoundry API key as `api_key`. The
  AzureOpenAI client sends it via the `api-key` header (not `Authorization:
      Bearer`) — the AI Gateway recognizes this natively, so no extra headers are
  needed. Your actual Azure credentials (API key, Entra ID, workload identity,
  etc.) stay configured on the Azure provider account in TrueFoundry and are
  never sent by the client.
</Note>

<Note>
  **Model is the TFY model FQN:** Pass the TrueFoundry model FQN
  (`provider-account/model-name`, for example `azure-openai-main/gpt-5-3-chat`)
  as `model` — this is what the AzureOpenAI client calls the "deployment name".
  The AI Gateway uses it as the routing identity and resolves the actual Azure
  deployment from your provider account configuration.
</Note>

Configure `AzureOpenAI` with `azure_endpoint` pointing to the AI Gateway's dedicated AzureOpenAI
sub-app: append `/azure-openai` to the AI Gateway base URL (which already includes `/api/llm` for
the standard TrueFoundry proxy):

```python lines theme={"dark"}
from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="your_truefoundry_api_key",
    azure_endpoint="{GATEWAY_BASE_URL}/azure-openai",
    api_version="2025-04-01-preview",
)

response = client.chat.completions.create(
    model="azure-openai-main/gpt-5-3-chat", # tfy model name
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)

print(response.choices[0].message.content)
```

<Note>
  **Why `/azure-openai`:** the AzureOpenAI SDK appends `/openai` to
  `azure_endpoint` itself — do not append `/openai` manually. Requests land on `   {GATEWAY_BASE_URL}/azure-openai/openai/...`, which the AI Gateway routes
  through a sub-app dedicated to Azure-SDK-compatible paths, keeping them
  isolated from the shared, provider-neutral endpoints (like the canonical
  `/responses` route) used by every other SDK on this page.
</Note>

<Note>
  Use `azure_endpoint`, not `base_url`. The gateway supports the AzureOpenAI
  deployment-scoped paths generated by `azure_endpoint`; `base_url` mode is not
  part of the supported AzureOpenAI surface.
</Note>

### Supported operations

* **Chat Completions** (`client.chat.completions.create`) — including streaming and tool calls
* **Completions** (`client.completions.create`)
* **Audio speech** (`client.audio.speech.create`)
* **Audio transcription** (`client.audio.transcriptions.create`)
* **Audio translation** (`client.audio.translations.create`)
* **Image generation** (`client.images.generate`)
* **Image edits** (`client.images.edit`)
* **Responses** (`client.responses.create`) — including streaming
* **Embeddings** (`client.embeddings.create`)

<Note>
  Files and batches are not currently supported through this deployment-scoped
  AzureOpenAI compatibility surface. They are account-level operations and do
  not carry the TFY model identity needed for gateway routing.
</Note>

### `api_version`

The `api_version` you pass to the client is used as-is for the upstream Azure request. If you omit it, the AI Gateway falls back to the `api_version` configured on the model in TrueFoundry. Azure still rejects versions it doesn't support for a given operation (for example, the Responses API requires `2025-03-01-preview` or later) regardless of which side supplied the version.

<Note>
  Deferred: the modern `OpenAI(base_url="{GATEWAY_BASE_URL}/openai/v1/")` client
  (Microsoft's recommended non-deployment-scoped surface) is not yet covered by
  this guide — use the `AzureOpenAI` client above.
</Note>

## boto3 SDK (AWS Bedrock Runtime)

Use `boto3` with the Bedrock Runtime client when you want Bedrock-native APIs (`converse`, `converse_stream`, `invoke_model`, `invoke_model_with_response_stream`) through the AI Gateway.

Set your endpoint to the AI Gateway Bedrock proxy and pass your TrueFoundry API key as `AWS_BEARER_TOKEN_BEDROCK`.

```python lines theme={"dark"}
import boto3
import os

os.environ["AWS_BEARER_TOKEN_BEDROCK"] = "your_truefoundry_api_key"

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime",
    region_name=os.environ.get("AWS_REGION", "dummy"),
    endpoint_url="{GATEWAY_BASE_URL}/bedrock/proxy",
)

response = bedrock_runtime.converse(
    modelId="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",  # tfy model name
    messages=[
        {"role": "user", "content": [{"text": "Hello, how are you?"}]},
    ],
    inferenceConfig={"maxTokens": 256, "temperature": 0.7},
)

for block in response["output"]["message"]["content"]:
    if "text" in block:
        print(block["text"], end="")
```

### Anthropic models with `invoke_model*`

For Anthropic models, Bedrock `invoke_model` and `invoke_model_with_response_stream` should use Anthropic Messages API fields:

* Add `anthropic_version` (for example, `bedrock-2023-05-31`)
* Add `max_tokens`
* Use typed content blocks: `{"type": "text", "text": "..."}`

Use typed blocks for `content` to avoid validation errors, for example: `messages.0.content.0.type: Field required`.

```python lines theme={"dark"}
import json

response = bedrock_runtime.invoke_model(
    modelId="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",
    contentType="application/json",
    accept="application/json",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 256,
        "messages": [
            {
                "role": "user",
                "content": [{"type": "text", "text": "Hello, how are you?"}],
            }
        ],
    }),
)
```

## LangChain AWS SDK

You can also use `langchain_aws` with the TFY AI Gateway Bedrock endpoint:

```python lines theme={"dark"}
from langchain_aws import ChatBedrockConverse

llm = ChatBedrockConverse(
    model="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",  # tfy model name
    region_name="dummy",
    endpoint_url="{GATEWAY_BASE_URL}/bedrock/proxy",
    api_key="your_truefoundry_api_key",
    temperature=0.2,
    max_tokens=256,
)

response = llm.invoke([("human", "Hello, how are you?")])
print(response.content)
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.