Proxy API (/proxy) - TrueFoundry Docs

Provider capabilities

The Proxy API (/proxy) forwards provider-native requests; coverage depends on which provider account and endpoint you use. It does not use a single consolidated provider matrix like the translated APIs. See Supported APIs for feature matrices on those endpoints. The Proxy API allows you to route requests directly to AI provider endpoints through TrueFoundry AI Gateway without any translation logic. This means you can use provider-native request and response formats while still benefiting from TrueFoundry’s features like logging, rate limiting, and budget management. TrueFoundry Proxy Endpoint routes your request to the appropriate provider and returns the provider’s response without modifying it, preserving the native request/response format.

Quick Start

Here’s a simple example to get you started immediately:

import openai

client = openai.OpenAI(
    api_key="your-truefoundry-api-key",
    base_url="{GATEWAY_BASE_URL}/proxy",
    default_headers={
        "x-tfy-provider-name": "openai-main"
    }
)

response = client.images.generate(
    model="dall-e-3",
    prompt="a futuristic cityscape at sunset with flying cars",
    n=1,
    size="1024x1024"
)

print("Image URL:", response.data[0].url)

About this example: This image generation request would work perfectly fine through TrueFoundry’s standard inference API. This example specifically demonstrates how to use the proxy route to make requests with provider-native formats while proxying requests via TrueFoundry AI Gateway.

That’s it! The proxy API works with your existing OpenAI SDK code. For more configuration options and other providers, see the detailed examples below.

Endpoint Structure

The proxy API supports multiple endpoint patterns for maximum flexibility:

Standard Proxy Endpoints

{GATEWAY_BASE_URL}/proxy/{provider-endpoint-path}

Model and Provider Configuration

The proxy API supports multiple ways to specify your model and provider name:

1. URL Query Parameters

tfyModelName - Complete TrueFoundry model name (e.g., provider-name/model-name)
tfyProviderName - TrueFoundry provider name for routing (only required when using actual model IDs)

2. Request Headers

x-tfy-model-name - Complete TrueFoundry model name
x-tfy-provider-name - TrueFoundry provider name (only required when using actual model IDs)

3. Request Body

model - Can be either TrueFoundry model name (provider-name/model-name) or actual model ID

Model and Provider Configuration Guide:When is Provider Name Required?

Not required: When using TrueFoundry model names (provider-name/model-name) - provider is auto-identified
Required: When using actual model IDs (gpt-4o, claude-3-sonnet) - specify provider via headers/query params

When is TrueFoundry Model Name Required?

Self-hosted models: Always required - base URL and configuration are model-specific
Full feature endpoints (Anthropic Messages, Claude Code, NIM Rerank): Required for logging, rate limiting, budget tracking
Other endpoints: Provider name alone is sufficient for basic routing

Model Name Processing:

TrueFoundry format (provider-name/model-name) → automatically converted to actual model ID
Actual model ID format → passed through unchanged to provider

Endpoint Support

Endpoints with Full Feature Support

These specific endpoints have complete logging, rate limiting, and budget management:

Anthropic Messages API - /messages
Claude Code - Available from Vertex, Bedrock, and Anthropic providers
NIM Rerank API - /ranking

Universal Proxy Support

You can use any provider endpoint through the TrueFoundry Proxy API, even if it doesn’t have specific feature support.

Code Examples

Endpoints with Full Feature Support

These endpoints provide complete TrueFoundry feature integration including logging, rate limiting, and budget management.

Anthropic Messages API

from anthropic import Anthropic

BASE_URL = "{GATEWAY_BASE_URL}/proxy"
API_KEY = "your-truefoundry-api-key"

# Configure Anthropic client with TrueFoundry settings
client = Anthropic(
    api_key=API_KEY,
    base_url=BASE_URL
)

# Make requests with your TrueFoundry model name in the request body
response = client.messages.create(
    model="anthropic-main/claude-3-5-sonnet-20241022",  # TrueFoundry model name
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Tell me a joke about programming"}
    ]
)

print(response.content)

import requests

url = "{GATEWAY_BASE_URL}/proxy/v1/messages"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}',
    "x-tfy-model-name": "anthropic-main/claude-3-5-sonnet-20241022"
}

payload = {
    "max_tokens": 1024,
    "messages": [
        {"role": "user", "content": "What are the benefits of renewable energy?"}
    ]
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

import requests

url = "{GATEWAY_BASE_URL}/proxy/v1/messages"
params = {
    "tfyModelName": "anthropic-main/claude-3-5-sonnet-20241022"
}

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "max_tokens": 1024,
    "messages": [
        {"role": "user", "content": "Explain quantum computing"}
    ]
}

response = requests.post(url, headers=headers, json=payload, params=params)
print(response.json())

NIM Rerank API

import requests

url = "{GATEWAY_BASE_URL}/proxy/v1/ranking"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "model": "custom-provider-main/llama-3.2-nemoretriever-500m-rerank-v2",
    "query": {
        "text": "What is machine learning?"
    },
    "passages": [
        {"text": "Machine learning is a subset of artificial intelligence"},
        {"text": "The weather is nice today"},
        {"text": "Python is a programming language"}
    ]
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

import requests

url = "{GATEWAY_BASE_URL}/proxy/v1/ranking"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}',
    "x-tfy-model-name": "custom-provider-main/llama-3.2-nemoretriever-500m-rerank-v2"
}

payload = {
    "query": {
        "text": "What is artificial intelligence?"
    },
    "passages": [
        {"text": "AI is a branch of computer science"},
        {"text": "Machine learning is a subset of AI"},
        {"text": "Today is a sunny day"}
    ]
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

import requests

url = "{GATEWAY_BASE_URL}/proxy/v1/ranking"
params = {
    "tfyModelName": "custom-provider-main/llama-3.2-nemoretriever-500m-rerank-v2"
}

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "query": {
        "text": "Where is New Delhi?"
    },
    "passages": [
        {"text": "New Delhi is capital of India"},
        {"text": "The sky is blue"},
        {"text": "Paris is the capital of France"}
    ]
}

response = requests.post(url, headers=headers, json=payload, params=params)
print(response.json())

Standard Proxy Examples

All provider endpoints can be proxied through TrueFoundry AI Gateway for basic routing, but won’t include advanced features like detailed logging, rate limiting, and budget management.

Custom Provider Endpoints

For self-hosted models, always provide the complete TrueFoundry model name since the base URL and configuration are model-specific.

import requests

# Using a self-hosted model endpoint
url = "{GATEWAY_BASE_URL}/proxy/{endpoint}"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "model": "my-custom-provider/my-model",  # TrueFoundry model name (will be auto-converted)
    "input": "Your custom payload here",
    "parameters": {
        "temperature": 0.7
    }
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

import requests

# Using a self-hosted model endpoint
url = "{GATEWAY_BASE_URL}/proxy/{endpoint}"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}',
    "x-tfy-model-name": "my-custom-provider/my-model"
}

payload = {
    "input": "Your custom payload here",
    "parameters": {
        "temperature": 0.7
    }
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

import requests

# Using a self-hosted model endpoint
url = "{GATEWAY_BASE_URL}/proxy/{endpoint}"
params = {
    "tfyModelName": "my-custom-provider/my-model"
}

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "input": "Your custom payload here",
    "parameters": {
        "temperature": 0.7
    }
}

response = requests.post(url, headers=headers, json=payload, params=params)
print(response.json())

Image Generation

For all other proxy endpoints: You can simply pass the TrueFoundry provider name via headers (x-tfy-provider-name) or query parameters (tfyProviderName) along with the actual model ID (if required), and the proxy will route your request correctly without requiring the full TrueFoundry model name format.

import openai

client = openai.OpenAI(
    api_key="your-truefoundry-api-key",
    base_url="{GATEWAY_BASE_URL}/proxy",
    default_headers={
        "x-tfy-provider-name": "openai-main"
    }
)

response = client.images.generate(
    model="dall-e-3",  # Actual model ID
    prompt="a serene mountain landscape with aurora borealis",
    n=1,
    size="1024x1024"
)

print("Image URL:", response.data[0].url)

import requests

url = "{GATEWAY_BASE_URL}/proxy/images/generations"
params = {
    "tfyProviderName": "openai-main"
}

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "X-TFY-LOGGING-CONFIG": '{"enabled": true}'
}

payload = {
    "model": "dall-e-3",
    "prompt": "a magical forest with glowing mushrooms and fairy lights",
    "n": 1,
    "size": "1024x1024"
}

response = requests.post(url, headers=headers, json=payload, params=params)
print(response.json())

Use Cases

This section demonstrates real-world applications and specific use cases for the Proxy API.

Mistral AI OCR - Document Processing

Extract text from documents while maintaining structure and formatting (headers, paragraphs, lists, tables). Returns markdown format and supports multiple formats including PDFs, images (png, jpeg, avif), and office documents (pptx, docx). Learn more about Mistral Document AI OCR capabilities.

import os
import base64
import requests
import json

def encode_pdf(pdf_path):
    with open(pdf_path, "rb") as pdf_file:
        return base64.b64encode(pdf_file.read()).decode("utf-8")

pdf_path = "path-to-your-pdf"
base64_pdf = encode_pdf(pdf_path)

url = "{GATEWAY_BASE_URL}/proxy/ocr"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-truefoundry-api-key",
    "x-tfy-provider-name": "mistral-provider-name"  # TrueFoundry Mistral provider name
}

payload = {
    "model": "mistral-ocr-latest",
    "document": {
        "type": "document_url",
        "document_url": f"data:application/pdf;base64,{base64_pdf}"
    },
    "include_image_base64": True
}

response = requests.post(url, headers=headers, json=payload)

if response.status_code == 200:
    # Save output to file
    with open("ocr_output.json", "w") as f:
        json.dump(response.json(), f, indent=2)
    print("OCR output saved to ocr_output.json")
else:
    print(f"Error: {response.status_code}")
    print(response.text)

​Provider capabilities

​Quick Start

​Endpoint Structure

​Standard Proxy Endpoints

​Model and Provider Configuration

​1. URL Query Parameters

​2. Request Headers

​3. Request Body

​Endpoint Support

​Endpoints with Full Feature Support

​Universal Proxy Support

​Code Examples

​Endpoints with Full Feature Support

​Anthropic Messages API

​NIM Rerank API

​Standard Proxy Examples

​Custom Provider Endpoints

​Image Generation

​Use Cases

​Mistral AI OCR - Document Processing

Provider capabilities

Quick Start

Endpoint Structure

Standard Proxy Endpoints

Model and Provider Configuration

1. URL Query Parameters

2. Request Headers

3. Request Body

Endpoint Support

Endpoints with Full Feature Support

Universal Proxy Support

Code Examples

Endpoints with Full Feature Support

Anthropic Messages API

NIM Rerank API

Standard Proxy Examples

Custom Provider Endpoints

Image Generation

Use Cases

Mistral AI OCR - Document Processing