Skip to main content

Documentation Index

Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt

Use this file to discover all available pages before exploring further.

Deploy the TrueFoundry control plane and AI Gateway. See the overview for compute requirements and prerequisites.

​
Installation Instructions

1

Create GCP Storage Bucket

We will need a GCS bucket to store the LLM request logs.
To setup GCP Storage for control plane, follow the steps below:
  1. Create a GCP bucket.
    • Make sure to add the lifecycle configurations on the bucket to delete multipart upload after 7 days.
    • For this go to GCP bucket -> Lifecycle -> Add a rule
    • Select Delete multi-part upload for 7 days
  2. We also need to add the CORS policy to the GCP bucket. Right now adding the CORS policy to the GCP bucket is not possible through the console so for this, we will use gsutil
    1. Create a file called cors.json using the below command
    cat > cors.json <<EOF
[
    {
      "origin": ["*"],
      "method": ["GET", "POST", "PUT"],
      "maxAgeSeconds": 3600
    }
]
EOF
  3. Attach the above CORS policy to the service account by running the following command using gsutils 
    gsutil cors set cors.json  gs://BUCKET_NAME
  4. Create a custom IAM role with the following permissions and add to the serviceaccount created above: 
    [
  "storage.objects.create",
  "storage.objects.delete",
  "storage.objects.get",
  "storage.objects.list",
  "storage.objects.update",
  "storage.buckets.create",
  "storage.buckets.get",
  "storage.buckets.list",
  "storage.buckets.create",
  "storage.buckets.update",
  "storage.multipartUploads.create",
  "storage.multipartUploads.list",
  "storage.multipartUploads.listParts",
  "storage.multipartUploads.abort",
  "resourcemanager.projects.get",
];
  5. Add the following IAM condition - resource.name.startsWith('projects/\_/buckets/<bucket name>}') to the above IAM role.
2

Create Google Cloud SQL for PostgreSQL database

Create a Google Cloud SQL for PostgreSQL database of size db-custom-1-3840 with storage size of 30GB.
Important Configuration Notes: - For PostgreSQL 17+: Disable SSL Security > Allow unencrypted network traffic - Firewall Rules: Ensure your database firewall rules allow inbound traffic from GKE node pools
In case you want to setup PostgreSQL on Kubernetes in the dev mode, skip this step and set devMode to true in the values file in the steps below.
3

Create Kubernetes Secrets

We will create two secrets in this step:
  1. Store the License Key and DB Credentials
  2. Store the Image Pull Secret
We need to create a Kubernetes secret containing the licence key and db credentials.
If you are using PostgreSQL on Kubernetes in the dev mode, the values will be as follows:DB_HOST: <HELM_RELEASE_NAME>-postgresql.<NAMESPACE>.svc.cluster.local // eg. truefoundry-postgresql.truefoundry.svc.cluster.localDB_NAME: truefoundryDB_USERNAME: postgres # In order to use custom username, please update the same at postgresql.auth.usernameDB_PASSWORD: randompassword # You can change this to any value here.
truefoundry-creds.yaml
apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-creds
type: Opaque
stringData:
  TFY_API_KEY: <TFY_API_KEY> # Provided by TrueFoundry team
  DB_HOST: <DB_HOST>
  DB_NAME: <DB_NAME>
  DB_USERNAME: <DB_USERNAME>
  DB_PASSWORD: <DB_PASSWORD>
Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the truefoundry namespace)
kubectl apply -f truefoundry-creds.yaml -n truefoundry
We need to create a Image Pull Secret to enable pulling the truefoundry images from the private registry.
truefoundry-image-pull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: truefoundry-image-pull-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <IMAGE_PULL_SECRET> # Provided by TrueFoundry team
Apply the secret to the Kubernetes cluster (Assuming you are installing the control plane in the truefoundry namespace)
kubectl apply -f truefoundry-image-pull-secret.yaml -n truefoundry
4

Create Helm chart Values file

Create a values file as given below and replace the following values:
  • Control Plane URL: URL that you will map to the control plane dashboard.
  • Tenant Name: Tenant name provided by TrueFoundry team.
  • GCP Project ID: GCP project ID provided by TrueFoundry team.
  • GCP Storage Bucket Name: Name of the GCS bucket you created in the previous step.
  • GCP Service Account Name: Name of the GCP service account you created in the previous step.
truefoundry-values.yaml
global:
  # Domain to map the control plane dashboard
  controlPlaneURL: https://example.com

  # Ask TrueFoundry team to provide these
  tenantName: <TENANT_NAME>

  # Choose the resource tier as per your needs
  resourceTier: medium # or small or large

  # This is the reference to the secrets we created in the previous step
  existingTruefoundryCredsSecret: "truefoundry-creds"
  imagePullSecrets:
    - name: "truefoundry-image-pull-secret"
  ## Add if you have restricted public registry access
  # image:
  #   pullSecretNames:
  #   - "truefoundry-image-pull-secret"

  config:
    defaultCloudProvider: "gcp"
    storageConfiguration:
      googleCloudProjectId: "<GCP_PROJECT_ID>"
      googleCloudStorageBucketName: "<GCP_STORAGE_BUCKET_NAME>"

  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: <CONTROL_PLANE_GCP_SERVICE_ACCOUNT_NAME>@<PROJECT_ID>.iam.gserviceaccount.com

  ingress:
    hosts:
      - example.com
    enabled: true
    annotations: {}
    ingressClassName: nginx # Replace with your ingress class name

# In case, you want to spin up PostgreSQL on kubernetes, enable this
devMode:
  enabled: false
tags:
  llmGateway: true
  llmGatewayRequestLogging: true

# Disable few dependencies for only LLM Gateway setup
tfyBuild:
  enabled: false
sfyManifestService:
  enabled: false
tfyController:
  enabled: false
tfyConfigs:
  enabled: false
5

Install Helm chart

helm upgrade --install truefoundry oci://tfy.jfrog.io/tfy-helm/truefoundry -n truefoundry --create-namespace -f truefoundry-values.yaml