Skip to main content
Applicable to: Customers using the TrueFoundry AI Engineering module who deploy applications (Service, Job, Notebook, Workflow, Spark Job, Volume, and related workloads) on a compute plane cluster.You must upgrade tfy-agent to version 0.2.95 or later on all compute plane clusters attached to your control plane before upgrading the control plane. If the control plane is upgraded first, new deployments will fail because the cluster lacks the ESO components needed to reconcile secrets.If you use only the AI Gateway module and not AI Engineering, this change does not apply to you.

​
What Is Changing

​
Before

Secret references (tfy-secret://...) in deployment specs were resolved to their actual values on the control plane, and those resolved values were embedded directly in the Kubernetes Secret objects within the ArgoCD Application manifests applied to your cluster. This meant the actual secret value ended up materialized in the Application YAML on the cluster — not ideal from a security standpoint. Updating a secret also required redeploying or re-syncing the application.

​
After

Starting with v0.151.5, TrueFoundry no longer puts secret values in the ArgoCD manifests. Instead, it adds an ExternalSecret custom resource to the deployment manifest. The External Secrets Operator (ESO) running on your cluster reads this resource, fetches the actual secret value from the control plane, and creates the Kubernetes Secret object locally. Here’s a simplified example of what the ExternalSecret resource looks like: 
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-image-pull-creds-external-secret
  labels:
    truefoundry.com/application: my-app
    truefoundry.com/component-type: service
spec:
  data:
    - remoteRef:
        key: tfy-image-pull-creds://<cluster>:<registry>
      secretKey: .dockerconfigjson
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: tfy-agent-secret-store
  target:
    name: my-app-image-pull-secret
    template:
      type: kubernetes.io/dockerconfigjson
ESO understands this resource and fetches the actual secret from the control plane to create the Kubernetes Secret. Secrets are refreshed every 1 hour by default.
Automatic secret refresh does not restart running pods. Pods will pick up the new secret values on their next restart. Automated pod restarts on secret rotation will be supported in a future release.
ESO is a Kubernetes operator that reads ExternalSecret custom resources and materializes them as standard K8s secret objects. TrueFoundry installs and configures ESO on your compute plane cluster through the tfy-agent Helm chart (version 0.2.95+).
ComponentScopeNamespaceRole
External Secrets Operator controllerClustertfy-agentWatches ExternalSecret resources and creates or updates K8s secret objects
ClusterSecretStore (tfy-secret-store)Cluster—Tells ESO how to reach the TrueFoundry control plane to resolve secret references
ExternalSecretPer applicationWorkload namespaceDeclares which TrueFoundry secret reference maps to which K8s secret object
K8s secret objectPer applicationWorkload namespaceThe in-cluster secret your Pods mount as env vars, volumes, or image pull credentials
The ClusterSecretStore uses a webhook provider that calls the TrueFoundry control plane:
{controlPlaneURL}/api/svc/v1/control-plane/secret?secret_ref={{ .remoteRef.key }}
The webhook authenticates using the cluster token stored in the tfy-agent token K8s secret object (CLUSTER_TOKEN key).

​
Affected workloads and secret types synced by ESO

Workload typeWhat is auto-synced
Service / Async ServiceEnvironment secrets, volume mounts, image pull credentials, and SQS autoscaling auth secrets
Job / Cron JobEnvironment and mount secrets, plus image pull credentials
Spark JobEnvironment and mount secrets, plus image pull credentials
WorkflowTask-level secrets and mounts
VolumeSecrets attached to volume operations
Workbench (Notebook, SSH Server)Environment and mount secrets
Alert ManagerNotification channel credentials (email, Slack, PagerDuty, and similar provider integrations)

​
Why This Change

  • Image pull secrets refresh automatically — Private registry credentials are also managed by ESO and refresh on the same interval, so updated registry tokens propagate without a full redeploy.
  • Single source of truth — Secret values are always resolved from the TrueFoundry control plane at runtime, matching the tfy-secret://... references in your deployment specs.
  • Standard Kubernetes pattern — ESO is the widely adopted operator for syncing external secrets into K8s. TrueFoundry manages its installation and configuration for you through tfy-agent.

​
What You Need to Do

1

Upgrade tfy-agent on every compute plane cluster

Update the tfy-agent Helm chart to version 0.2.95 or later on each compute plane cluster. This version includes the External Secrets Operator subchart and the ClusterSecretStore configuration. See the values reference for all available options.Verify ESO is running after the upgrade:
kubectl get pods -n tfy-agent -l app.kubernetes.io/name=external-secrets
kubectl get clustersecretstore tfy-secret-store
The ClusterSecretStore should report a Ready status once the cluster token K8s secret object is configured correctly.
2

Add the webhook label if you use an external cluster token secret

If you pass the cluster token inline via config.clusterToken in the tfy-agent values file, the chart creates the token K8s secret object and adds the required label automatically when ESO is enabled. No action needed.If you store the cluster token in an existing K8s secret object referenced by config.clusterTokenSecret, you must add this label so ESO can use it as a webhook authentication source:
metadata:
  labels:
    external-secrets.io/type: webhook
The K8s secret object must contain a key named CLUSTER_TOKEN. Find the secret name in your tfy-agent values file under config.clusterTokenSecret.Example using kubectl:
kubectl label secret <your-cluster-token-secret-name> \
  -n tfy-agent \
  external-secrets.io/type=webhook
Replace <your-cluster-token-secret-name> with the value of config.clusterTokenSecret from your tfy-agent Helm values.
3

Confirm ESO is enabled in tfy-agent values

ESO is enabled by default. Confirm these settings in your tfy-agent values file (or rely on the defaults):
external-secrets-operator:
  enabled: true
  clusterSecretStore:
    create: true
    name: tfy-secret-store
If you already run External Secrets Operator in your cluster, set external-secrets-operator.enabled: false to avoid installing a second ESO instance (running two operators can cause them to fight over the same resources). With the subchart disabled, tfy-agent still creates the ClusterSecretStore (tfy-secret-store) that TrueFoundry’s ExternalSecret resources reference, and your existing operator reconciles them.
external-secrets-operator:
  enabled: false
  clusterSecretStore:
    create: true
    name: tfy-secret-store
Your existing ESO installation must be a compatible version that supports the external-secrets.io/v1 API and the webhook provider.
See the full tfy-agent values reference for all available options.
4

Redeploy or sync affected applications

After tfy-agent is upgraded and the cluster token K8s secret object is labeled (if applicable), redeploy or sync applications that use secrets. New manifest generation produces ExternalSecret resources that ESO uses to manage K8s secret objects.You can verify on a running application:
kubectl get externalsecret -n <workspace-namespace>

kubectl get secret -n <workspace-namespace> -l app.kubernetes.io/managed-by=external-secrets

​
Configuration reference

For the complete list of ESO-related settings and their defaults, see the external-secrets-operator section of the tfy-agent values file.
If you have questions or run into issues during the migration, reach out to support@truefoundry.com — we’re happy to help.