Skip to main content
LlamaIndex is an open-source data framework for building applications with large language models (LLMs). TrueFoundry seamlessly integrates with LlamaIndex, allowing you to route all LLM requests through the TrueFoundry Gateway for enhanced security, routing, cost management, and more. This guide will walk you through connecting LlamaIndex with TrueFoundrry.
You should only use the OpenAILike and OpenAIEmbedding classes from LlamaIndex. These classes are designed for custom OpenAI-compatible endpoints, allowing you to use your TrueFoundry-specific model names directly. Using the standard OpenAI class will cause errors, as it validates model names against OpenAI’s public list.

​
Prerequisites

Before you begin, ensure you have the following:
  1. Authentication Token: A TrueFoundry API key. Follow the instructions in Generating Tokens to create one.
  2. Gateway Base URL: Your TrueFoundry Gateway Base URL ({GATEWAY_BASE_URL}). See Authentication for details.

​
Code Examples

The following examples demonstrate how to use LlamaIndex with TrueFoundry.

​
Chat Completion

This example shows how to perform chat completion. 
from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike

# Configure the LLM
llm = OpenAILike(
    model="tfy_model_name",
    api_key="tfy_api_key",
    api_base="{GATEWAY_BASE_URL}",
    is_chat_model=True
)

# Set the LLM for LlamaIndex to use globally
Settings.llm = llm

# Perform a chat completion
response = llm.complete("What is the capital of France?")
print(response.text)

​
Text Embedding

This example shows how to generate text embeddings. 
from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding

# Configure the LLM
embedding_model = OpenAIEmbedding(
    model_name="tfy_model_name",
    api_key="tfy_api_key",
    api_base="{GATEWAY_BASE_URL}",
)

# Set the embedding model for LlamaIndex to use globally
Settings.embed_model = embedding_model

# Generate a text embedding
response = embedding_model.get_text_embedding("This is a sample text.")
print(f"Embedding length: {len(response)}")

​
FAQs

This error usually occurs when you’re using the OpenAI class from LlamaIndex instead of OpenAILike.
The OpenAI class strictly validates model names against OpenAI’s public list, so any custom or TrueFoundry-specific model name will result in this error.
To fix this, make sure you are using the OpenAILike class, which supports custom, OpenAI-compatible endpoints.