Skip to main content

​
Key Components

To install only AI Gateway on your own infrastructure, you need the following components:
  • Truefoundry AI Gateway (Shipped as a single helm chart called truefoundry)
  • Blob Storage (S3, GCS, Azure Container or any other S3 compatible storage)

​
Prerequisites for Installation

  1. Kubernetes Cluster: K8s cluster 1.27+.
  2. Support for dynamic provisioning of storage for PVC (for e.g AWS EBS, Azure Disk etc.) and support for ingress controller (for e.g. Nginx Ingress Controller) or istio service mesh for exposing the control plane dashboard and AI Gateway at an endpoint.
  3. Domain to map the ingress of the AI Gateway API server along with certificate for the domain.
    This Domain will be referred as Gateway URL in our documentation.
  4. Egress access to TrueFoundry Central Auth Server: https://auth.truefoundry.com & https://login.truefoundry.com
  5. Tenant Name, Control Plane URL, Licence key, and image pull secret from TrueFoundry team. If you have not registered yet, please visit TrueFoundry to register.
  6. Blob Storage to store the AI Gateway request logs (either S3, GCS, Azure Blob Storage, or any other S3 compatible storage). You can find the instructions in the guide below.

​
Installation Instructions

1

Setup AI Gateway IAM Role

AI Gateway IAM Role needs to have permission to assume any other IAM role in or cross account to provide access to different cloud services like bedrock models, etc.
  • Create a new IAM role for AI Gateway with a suitable name like tfy-ai-gateway-deps
  • Following is the IAM policy that needs to be attached to the AI Gateway IAM Role:
{
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}
Here ”*” is used to allow the AI Gateway IAM Role to assume any other IAM role in or cross account. In place of ”*” you can also give specific ARNs of other IAM roles
  • Add the following trust policy to the AI Gateway IAM Role:
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
              "StringEquals": {
                  "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": [
                      "system:serviceaccount:truefoundry:truefoundry",
                  ]
              }
          }
      }
  ]
}
In place of <ACCOUNT_ID>, <AWS_REGION>, and <OIDC_ID> you can also give the values from your EKS cluster. You can find the OIDC_ID from the EKS cluster. Also, here we are assuming that the service account is truefoundry and the namespace is truefoundry, you can change it as per your needs.

​
Allow your IAM role to be assumed by AI Gateway IAM Role

  • Add the following trust policy to your IAM Role to allow it to be assumed by the AI Gateway IAM Role:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<AI_GATEWAY_IAM_ROLE_ARN>"
            },
            "Action": "sts:AssumeRole"
        }
      ]
}
2

Create S3 Bucket

Create a S3 Bucket with following config:
  • Make sure the bucket has lifecycle configuration to abort multipart upload set for 7 days.
  • Make sure CORS is applied on the bucket with the below configuration:
[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]
Create a IAM Policy to allow access to the S3 Bucket with following config:
{
  "Sid": "S3",
  "Effect": "Allow",
  "Action": ["s3:*"],
  "Resource": [
    "arn:aws:s3:::<YOUR_S3_BUCKET_NAME>",
    "arn:aws:s3:::<YOUR_S3_BUCKET_NAME>/*"
  ]
}
Attach the IAM Policy to the AI Gateway IAM Role
3

Create Kubernetes Secrets

We will create two secrets in this step:
  1. Store the License Key
  2. Store the Image Pull Secret
We need to create a Kubernetes secret containing the licence key.
truefoundry-creds.yaml
apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-creds
type: Opaque
stringData:
  TFY_API_KEY: <TFY_API_KEY> # Provided by TrueFoundry team
Apply the secret to the Kubernetes cluster (Assuming you are installing the AI gateway in the truefoundry namespace)
kubectl apply -f truefoundry-creds.yaml -n truefoundry
We need to create a Image Pull Secret to enable pulling the truefoundry images from the private registry.
truefoundry-image-pull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-image-pull-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <IMAGE_PULL_SECRET> # Provided by TrueFoundry team
Apply the secret to the Kubernetes cluster (Assuming you are installing the ai gateway in the truefoundry namespace)
kubectl apply -f truefoundry-image-pull-secret.yaml -n truefoundry
4

Create HelmChart Values file

Create a values file as given below and replace the following values:
  • Control Plane URL: Control Plane URL provided by TrueFoundry team (e.g., https://truefoundry.cp.example.com)
  • Tenant Name: Tenant name provided by TrueFoundry team
  • AI Gateway Host: URL that you will map to your AI Gateway API Server (e.g., truefoundry.gateway.example.com)
  • AWS S3 Bucket Name: Name of the S3 bucket you created in the previous step (e.g., my-truefoundry-bucket)
  • AWS Region: Region of the S3 bucket you created in the previous step (e.g., us-west-2)
  • AI Gateway IAM Role ARN: ARN of the IAM role you created in the previous step (e.g., arn:aws:iam::123456789012:role/tfy-ai-gateway-deps)
truefoundry-values.yaml
global:
  # URL of the SaaS control plane. Ask TrueFoundry team to provide these.
  controlPlaneURL: https://truefoundry.cp.example.com

  # Ask TrueFoundry team to provide these.
  tenantName: <TENANT_NAME>

  # Choose the resource tier as per your needs
  resourceTier: medium # or small or large

  # This is the reference to the secrets we created in the previous step
  existingTruefoundryCredsSecret: "truefoundry-creds"
  imagePullSecrets:
    - name: "truefoundry-image-pull-secret"
  ## Add if you have restricted public registry access
  # image:
  #   pullSecretNames:
  #   - "truefoundry-image-pull-secret"

  config:
    defaultCloudProvider: "aws"
    storageConfiguration:
      awsS3BucketName: "<AWS_S3_BUCKET_NAME>"
      awsRegion: "<AWS_REGION>"

  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: <AI_GATEWAY_IAM_ROLE_ARN>
tags:
  llmGateway: true
  llmGatewayRequestLogging: true

tfy-llm-gateway:
  ingress:
    hosts:
      - <AI_GATEWAY_HOST>
    enabled: true
    annotations: {}
    ingressClassName: nginx # Replace with your ingress class name
  env:
    CONTROL_PLANE_NATS_URL: wss://{{ .Values.global.controlPlaneURL | trimPrefix "https://" }}
    SERVICEFOUNDRY_SERVER_URL: {{ .Values.global.controlPlaneURL }}/api/svc

tfy-otel-collector:
  env:
    SFY_SERVER_BASE_URL: "{{ .Values.global.controlPlaneURL }}/api/svc"
deltaFusionIngestor:
  env:
    SERVICEFOUNDRY_SERVER_URL: "{{ .Values.global.controlPlaneURL }}/api/svc"

# Disable few dependencies for only LLM Gateway setup
tfyNats:
  enabled: false
tfy-clickhouse:
  enabled: false
truefoundryFrontendApp:
  enabled: false
servicefoundryServer:
  enabled: false
tfyProxy:
  enabled: false
tfyK8sController:
  enabled: false
truefoundryBootstrap:
  enabled: false
tfyBuild:
  enabled: false
tfy-buildkitd-service:
  enabled: false
mlfoundryServer:
  enabled: false
sfyManifestService:
  enabled: false
tfyController:
  enabled: false
deltaFusionCompaction:
  enabled: false
deltaFusionQueryServer:
  enabled: false
5

Install Helm chart

helm upgrade --install truefoundry oci://tfy.jfrog.io/tfy-helm/truefoundry -n truefoundry --create-namespace -f truefoundry-values.yaml
6

Provide Access to TrueFoundry Control Plane

Provide following information to TrueFoundry team on setup completion:
  1. AI Gateway URL configured to get it whitelisted on TrueFoundry side.
  2. AWS Access Key ID, AWS Secret Key, Region and Bucket name for accessing the Bucket to show metrics and request logs in the Control Plane Dashboard.