Deploy AI Gateway FAQ - TrueFoundry Docs

How to add multiple gateway planes to the control plane?

You can add multiple gateway planes to the control plane by following the steps below:

Create Kubernetes Secret for License Key and DB Credentials

We will create two secrets in this step:

Store the License Key
Store the Image Pull Secret

Create Kubernetes Secret for License Key

We need to create a Kubernetes secret containing the licence key.

Same license key will be used for all the gateway planes as used for the control plane

truefoundry-creds.yaml

apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-creds
type: Opaque
stringData:
  TFY_API_KEY: <TFY_API_KEY>

Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the truefoundry namespace)

kubectl apply -f truefoundry-creds.yaml -n truefoundry

Create Kubernetes Secret for Image Pull Secret

We need to create a Image Pull Secret to enable pulling the truefoundry images from the private registry.

Same image pull secret will be used for all the gateway planes as used for the control plane. Use your credentials if you are pulling TrueFoundry images from your registry.

truefoundry-image-pull-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-image-pull-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <IMAGE_PULL_SECRET> # Provided by TrueFoundry team

Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the truefoundry namespace)

kubectl apply -f truefoundry-image-pull-secret.yaml -n truefoundry

Create Helm chart Values file for gateway plane

Create a values file as given below and replace the following values:

CONTROL_PLANE_URL: URL that you will map to the control plane dashboard.
TENANT_NAME: Tenant name provided by TrueFoundry team.
GATEWAY_ENDPOINT_HOST: The domain where you will expose the gateway endpoint (e.g., gateway.example.com)

truefoundry-gateway-values.yaml

global:
  # This is the reference to the secrets we created in the previous step
  imagePullSecrets:
    - name: "truefoundry-image-pull-secret"

  # Choose the resource tier as per your needs
  resourceTier: medium # or small or large
  controlPlaneURL: <CONTROL_PLANE_URL> # eg. https://example-company.truefoundry.cloud
  tenantName: <TENANT_NAME>

ingress:
  enabled: true
  annotations: {}
  ingressClassName: nginx
  tls: []
  hosts:
    - <GATEWAY_ENDPOINT_HOST>

# Optional: Istio configuration (if using Istio instead of standard ingress)
# istio:
#   virtualservice:
#     hosts:
#       - <GATEWAY_ENDPOINT_HOST>
#     enabled: true
#     retries:
#       enabled: true
#       retryOn: gateway-error
#     gateways:
#       - istio-system/tfy-wildcard
#     annotations: {}

Install Helm chart for gateway plane

helm upgrade --install tfy-llm-gateway oci://tfy.jfrog.io/tfy-helm/tfy-llm-gateway -n truefoundry --create-namespace -f truefoundry-gateway-values.yaml

Can I use my Artifactory as a mirror to pull images?

Yes. You can configure your Artifactory to mirror our registry.

Credentials for accessing the TrueFoundry private registry are required and will be provided during onboarding.

1. Registry Configuration

URL: https://tfy.jfrog.io/

2. Update Helm values

global:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry
postgresql:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry, use this if `devMode` is enabled

Can I copy images to my own private registry?

Yes. We provide a script that uses the truefoundry Helm Chart to identify and copy required images to your private registry.

Credentials for accessing the TrueFoundry private registry are required and will be provided during onboarding.

Generic Registry
AWS ECR Registry

1. Install required dependencies

Skopeo
- Used to perform the image copy operation.
Helm
- Used to get the list of images from the TrueFoundry Helm Chart.

2. Add TrueFoundry Helm Chart repository

helm repo add truefoundry https://truefoundry.github.io/infra-charts
helm repo update

3. Authenticate to the TrueFoundry source registry

skopeo login -u <USERNAME> -p <PASSWORD> https://tfy.jfrog.io/

Replace <USERNAME> with the TrueFoundry registry username.
Replace <PASSWORD> with the TrueFoundry registry password.

4. Authenticate to your destination registry

skopeo login -u <USERNAME> -p <PASSWORD> <YOUR_REGISTRY>

Replace <USERNAME> with your registry username.
Replace <PASSWORD> with your registry password.
Replace <YOUR_REGISTRY> with the URL of your registry.Skopeo will use authentication details for a registry that was previously authenticated with docker login.Alternatively, you can use the --dest-user and --dest-password flags to provide the username and password for the destination registry.

5. Run Clone Image Script

export TRUEFOUNDRY_HELM_CHART_VERSION=<TRUEFOUNDRY_HELM_CHART_VERSION>
export TRUEFOUNDRY_HELM_VALUES_FILE=<TRUEFOUNDRY_HELM_VALUES_FILE>
export DEST_REGISTRY=<YOUR_DESTINATION_REGISTRY>

# Dry-run example
curl -s https://raw.githubusercontent.com/truefoundry/infra-charts/main/scripts/clone_images_to_your_registry.sh | bash -s -- --helm-chart truefoundry --helm-version $TRUEFOUNDRY_HELM_CHART_VERSION --helm-values $TRUEFOUNDRY_HELM_VALUES_FILE --dest-registry $DEST_REGISTRY --dry-run

# Live example
curl -s https://raw.githubusercontent.com/truefoundry/infra-charts/main/scripts/clone_images_to_your_registry.sh | bash -s -- --helm-chart truefoundry --helm-version $TRUEFOUNDRY_HELM_CHART_VERSION --helm-values $TRUEFOUNDRY_HELM_VALUES_FILE --dest-registry $DEST_REGISTRY

Replace <TRUEFOUNDRY_HELM_CHART_VERSION> with the version of the Truefoundry helm chart you want to use. You can find the latest version in the changelog.Replace <TRUEFOUNDRY_HELM_VALUES_FILE> with the path to the values file you created in the Installation Instructions.Replace <DEST_REGISTRY> with the URL of your registry.

6. Update the Helm values file to use your registry

global:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry
postgresql:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry, use this if `devMode` is enabled

1. Install required dependencies

Skopeo
- Used to perform the image copy operation.
Helm
- Used to get the list of images from the TrueFoundry Helm Chart.
AWS CLI
- Used to perform AWS ECR actions to validate and create repositories.

2. Add TrueFoundry Helm Chart repository

helm repo add truefoundry https://truefoundry.github.io/infra-charts
helm repo update

3. Authenticate to the TrueFoundry source registry

skopeo login -u <USERNAME> -p <PASSWORD> https://tfy.jfrog.io/

Replace <USERNAME> with the TrueFoundry registry username.
Replace <PASSWORD> with the TrueFoundry registry password.

4. Authenticate to your destination registry

# Set your AWS profile
export AWS_PROFILE=<AWS_PROFILE>

# Authenticate to ECR using the profile
aws ecr get-login-password --region us-west-2 | skopeo login --username AWS --password-stdin <YOUR_ECR_REGISTRY>

Replace <AWS_PROFILE> with your AWS profile name.
Replace <YOUR_ECR_REGISTRY> with the URL of your ECR registry (ex. 123456789012.dkr.ecr.us-east-2.amazonaws.com).Skopeo will use authentication details for a registry that was previously authenticated with docker login.

5. Run Clone Image Script

This script creates required ECR repositories and copies images.
Optionally append a path to your registry URL to namespace repositories (e.g., 123456789012.dkr.ecr.us-east-2.amazonaws.com/truefoundry).

export TRUEFOUNDRY_HELM_CHART_VERSION=<TRUEFOUNDRY_HELM_CHART_VERSION>
export TRUEFOUNDRY_HELM_VALUES_FILE=<TRUEFOUNDRY_HELM_VALUES_FILE>
export DEST_REGISTRY=<YOUR_DESTINATION_REGISTRY>

# Dry-run example
curl -s https://raw.githubusercontent.com/truefoundry/infra-charts/main/scripts/clone_images_to_your_registry.sh | bash -s -- --helm-chart truefoundry --helm-version $TRUEFOUNDRY_HELM_CHART_VERSION --helm-values $TRUEFOUNDRY_HELM_VALUES_FILE --dest-registry $DEST_REGISTRY --dry-run

# Live example
curl -s https://raw.githubusercontent.com/truefoundry/infra-charts/main/scripts/clone_images_to_your_registry.sh | bash -s -- --helm-chart truefoundry --helm-version $TRUEFOUNDRY_HELM_CHART_VERSION --helm-values $TRUEFOUNDRY_HELM_VALUES_FILE --dest-registry $DEST_REGISTRY

Replace <TRUEFOUNDRY_HELM_CHART_VERSION> with the TrueFoundry Helm chart version. Find the latest version in the changelog.Replace <TRUEFOUNDRY_HELM_VALUES_FILE> with the path to your values file from Installation Instructions.Replace <YOUR_DESTINATION_ECR_REGISTRY> with your ECR registry URL (e.g., 123456789012.dkr.ecr.us-east-2.amazonaws.com/truefoundy).

6. Update the Helm values file to use your registry

global:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry
postgresql:
  image:
    registry: <YOUR_REGISTRY> # Replace with your registry, use this if `devMode` is enabled

How to install in an air-gapped / restricted network environment?

An air-gapped environment is isolated from the internet. Since the control plane and gateway plane ship as a single helm chart (truefoundry), you only need to make the container images available in your private registry and update the helm values to point to it.

Copy images to your private registry — set up a registry mirror or copy images directly using the steps described in the FAQs above
Update helm values to point to your private registry (see the helm value overrides in the same FAQs above)
Continue with the standard installation on the overview and choose your cloud install guide (AWS, GCP, Azure, or on-prem)

How to integrate with AWS bedrock models from a different AWS account?

You can integrate with AWS bedrock models from a different AWS account by following the steps below:

Add the following IAM policy to the control plane IAM role so that it can assume the IAM role of the AWS account that has the bedrock models:

{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Resource": "*"
    }
  ],
  "Version": "2012-10-17"
}

In the IAM role in the destination AWS account (which has bedrock access), add the following trust policy to allow the control plane IAM role to assume it:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<CONTROL_PLANE_IAM_ROLE_ARN>"
      },

      "Action": "sts:AssumeRole"
    }
  ],
  "Version": "2012-10-17"
}

Now you can use the IAM role of the destination AWS account while integrating AWS bedrock models in the TrueFoundry AI gateway.

Do we need any NFS volumes in Kubernetes for the AI Gateway or Control Plane?

No, we only need block storage for installing and running Truefoundry. This should be supported via the CSI driver and only ReadWriteOnce access is required.

What is the structure of access logs

We log access information in standard output with the following format:

logfmt
json

These can be switched with the help of an environment variable to the AI Gateway installation. (Default: logfmt)

Log format

Standard log format structure:

time="%START_TIME%" level=%LEVEL% ip=%IP_ADDRESS% tenant=%TENANT_NAME% user=%SUBJECT_TYPE%:%SUBJECT_SLUG% model=%MODEL_ID% method=%METHOD% path=%PATH% status=%STATUS_CODE% time_taken=%DURATION%ms trace_id=%TRACE_ID%

Log operator	Details
START_TIME	ISO timestamp for request start. eg. 2025-08-12 13:34:50
LEVEL	info\|warn\|error
IP_ADDRESS	IP address of the caller. eg. ::ffff:10.99.55.142
TENANT_NAME	Name of the tenant. eg. truefoundry
SUBJECT_TYPE	user\|virtualaccount
SUBJECT_SLUG	Email or virtual account name. eg. tfy-user@truefoundry.com\|demo-virtualaccount
MODEL_ID	Model ID. eg. openai-default/gpt-5
METHOD	GET\|POST\|PUT
PATH	Path of the request. eg. /api/inference/openai/chat/completions
STATUS_CODE	200\|400\|401\|403\|429\|500
DURATION	Duration of the request. eg. 12
TRACE_ID	Trace ID of the request

Examples:

time="2025-08-12 13:34:50" level=info ip=::ffff:10.99.55.142 tenant=truefoundry user=virtualaccount:demo-virtualaccount model=openai-default/gpt-5 method=POST path=/api/inference/openai/chat/completions status=200 time_taken=53ms trace_id=587b2a946c13f62f9160674a8c983ce3

How to use SSO directly without using TrueFoundry Auth Server?

By default, the control plane uses the TrueFoundry Auth Server for user authentication. However, you can configure it to use your own external identity provider instead. We support both OIDC and SAML-compliant identity providers. Read more

Requests to the gateway are timing out after a certain duration

If your LLM requests are timing out after a certain duration, the first thing to check is the traces in the TrueFoundry dashboard. Look at the request duration — if you see requests consistently timing out at exactly 60 seconds, the issue is almost certainly the load balancer, not the TrueFoundry AI Gateway. The TrueFoundry gateway does not impose any request timeout.

Traces showing requests timing out at 60 seconds

This commonly happens when an Application Load Balancer (ALB) is placed in front of the gateway to expose it. The default Connection idle timeout on AWS ALBs is 60 seconds, which is too short for long-running LLM inference requests (especially streaming responses or large prompts).Solution: Increase the idle timeout on your AWS ALB to a higher value (e.g., 300 seconds or more).You can find this setting in the AWS Console under EC2 → Load Balancers → Select your ALB → Attributes tab → Connection idle timeout.

You can also update it via the AWS CLI:

aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn <YOUR_ALB_ARN> \
  --attributes Key=idle_timeout.timeout_seconds,Value=300

If you are using an ingress controller (e.g., NGINX Ingress) in addition to the ALB, also verify that the ingress controller’s proxy timeout settings are configured appropriately.

Can I get TrueFoundry metrics in Victoria Metrics instead of Prometheus?

Yes. TrueFoundry supports exporting metrics to Victoria Metrics as an alternative to Prometheus. To enable this, add the following to your truefoundry-values.yaml file and upgrade the Helm release:

This only installs the VMServiceScrape and related custom resources for scraping TrueFoundry metrics. It does not deploy Victoria Metrics itself — you are responsible for installing and managing your own Victoria Metrics instance.

truefoundry-values.yaml

victoriaMetricsMonitoring:
  enabled: true

Then upgrade the Helm release to apply the changes:

helm upgrade --install truefoundry oci://tfy.jfrog.io/tfy-helm/truefoundry -n truefoundry --create-namespace -f truefoundry-values.yaml

Platform

Documentation Index

​FAQ

​Log format

FAQ

Log format