Skip to main content
Applicable to: All users deploying applications (Service, Job, Notebook, Workflow, Spark Job, Volume, and related workloads) on a TrueFoundry compute plane cluster. Requires tfy-agent chart version 0.2.94 or later on every compute plane cluster where ESO-based secret is enabled.
TrueFoundry deploys workloads to your compute plane through ArgoCD. Until now, secret values were rendered into Kubernetes Secret objects in the generated Helm manifests and synced by ArgoCD like any other resource. With External Secrets Operator (ESO), ArgoCD only syncs ExternalSecret specifications that reference secrets in TrueFoundry. ESO fetches the actual values directly from the control plane and creates the in-cluster Secret objects. This keeps secret material out of the ArgoCD sync path while preserving the same developer experience — you still reference secrets with tfy-secret://... FQNs in deployment specs.

​
What is changing

​
Before: secrets synced through ArgoCD

When you deploy a Service, Job, Notebook, Workflow, or similar workload, TrueFoundry generates Helm manifests on the control plane and ArgoCD applies them to the compute plane cluster. In t he previous flow, sensitive values were resolved on the control plane and written into Kubernetes Secret objects inside those manifests. In this model, secret values travel through the same ArgoCD application sync that deploys Deployments, Services, and other resources.

​
After: secrets delivered by External Secrets Operator

TrueFoundry now generates ExternalSecret custom resources instead of in-cluster Secret objects when ESO is enabled. Each ExternalSecret contains only a reference (for example, tfy-secret://workspace:group:key) — not the secret value itself. ArgoCD syncs the ExternalSecret spec; ESO reads it, calls the TrueFoundry control plane to resolve the reference, and creates the target Kubernetes Secret in the workload namespace. Secret values no longer pass through ArgoCD. ArgoCD only manages the ExternalSecret declaration; ESO handles secret materialization.

​
End-to-end request flow

The following sequence shows what happens when a deployed application needs a secret referenced as tfy-secret://my-workspace:api-keys:API_KEY.

​
Why this change

  • Reduced secret exposure in ArgoCD — ArgoCD no longer receives or stores resolved secret values in the manifests it syncs. Only opaque references (tfy-secret://..., tfy-image-pull-creds://..., and similar) appear in the sync path.
  • Consistent secret resolution — Secrets are fetched at runtime from the TrueFoundry control plane, the same source of truth you use when authoring deployment specs.
  • Automatic refresh — ESO periodically re-fetches secrets (default refresh interval: 1h), so rotated values can propagate to running workloads without redeploying through ArgoCD.
  • Alignment with Kubernetes best practices — External Secrets Operator is the standard pattern for decoupling secret storage from deployment configuration managed by ArgoCD.

​
What TrueFoundry deploys on the compute plane

ESO is installed as part of the tfy-agent Helm chart, which runs on every compute plane cluster and connects that cluster to the control plane. Starting in tfy-agent chart version 0.2.94, the chart bundles the upstream external-secrets operator as a subchart. When external-secrets-operator.enabled is true (the default), tfy-agent also creates a cluster-scoped ClusterSecretStore named tfy-secret-store (configurable via external-secrets-operator.clusterSecretStoreName). This store is configured with a webhook provider that calls the TrueFoundry control plane: 
{controlPlaneURL}/api/svc/v1/control-plane/secret?secret_ref={{ .remoteRef.key }}
The webhook authenticates using the cluster token stored in the tfy-agent token secret (CLUSTER_TOKEN key). All ExternalSecret resources generated by TrueFoundry reference this ClusterSecretStore.
tfy-agent must be able to reach the control plane URL configured in config.controlPlaneURL. The same outbound connectivity used by tfy-agent today is sufficient for ESO secret resolution.

​
Affected workloads and secret types

ESO support covers secrets used across TrueFoundry deployment types, including:
Workload typeWhat changes
Service / Async ServiceEnvironment secrets, volume mounts, image pull credentials, and SQS autoscaling auth secrets are delivered via ExternalSecret
Job / Cron JobEnvironment and mount secrets, plus image pull credentials
Spark JobEnvironment and mount secrets, plus image pull credentials
WorkflowTask-level secrets and mounts
VolumeSecrets attached to volume operations
Workbench (Notebook, SSH Server)Environment and mount secrets
Alert ManagerNotification channel credentials (email, Slack, PagerDuty, and similar provider integrations)

​
What you need to do

1

Upgrade tfy-agent on every compute plane cluster

Update the tfy-agent Helm chart to version 0.2.94 or later on each compute plane cluster. This version includes the External Secrets Operator subchart and the ClusterSecretStore configuration.Verify ESO is running after the upgrade:
kubectl get pods -n tfy-agent -l app.kubernetes.io/name=external-secrets
kubectl get clustersecretstore tfy-secret-store
The ClusterSecretStore should report a Ready status once the cluster token secret is configured correctly.
2

Add the webhook label if you use an external cluster token secret

If you pass the cluster token inline via config.clusterToken in the tfy-agent values file, the chart creates the token secret and adds the required label automatically when ESO is enabled.If you store the cluster token in an existing Kubernetes Secret referenced by config.clusterTokenSecret, you must add this label to that secret so ESO can use it as a webhook authentication source:
metadata:
  labels:
    external-secrets.io/type: webhook
The secret must contain a key named CLUSTER_TOKEN. Find the secret name in your tfy-agent values file under config.clusterTokenSecret.Example using kubectl:
kubectl label secret <your-cluster-token-secret-name> \
  -n tfy-agent \
  external-secrets.io/type=webhook
Replace <your-cluster-token-secret-name> with the value of config.clusterTokenSecret from your tfy-agent Helm values.
3

Confirm ESO is enabled in tfy-agent values

ESO is enabled by default. Confirm these settings in your tfy-agent values file (or rely on the defaults):
external-secrets-operator:
  enabled: true
  clusterSecretStoreName: tfy-secret-store
See the full tfy-agent values reference for all available options.
4

Redeploy or sync affected applications

After tfy-agent is upgraded and the cluster token secret is labeled (if applicable), redeploy or sync applications that use secrets. New manifest generation produces ExternalSecret resources instead of inline Secret objects.You can verify the transition on a running application:
# ExternalSecret should exist in the application namespace
kubectl get externalsecret -n <workspace-namespace>

# Target Secret is created by ESO, not ArgoCD
kubectl get secret -n <workspace-namespace> -l app.kubernetes.io/managed-by=external-secrets

​
Configuration reference

SettingLocationDefaultDescription
external-secrets-operator.enabledtfy-agent Helm valuestrueInstalls ESO and creates the ClusterSecretStore
external-secrets-operator.clusterSecretStoreNametfy-agent Helm valuestfy-secret-storeName of the ClusterSecretStore referenced by application ExternalSecret resources
config.clusterTokenSecrettfy-agent Helm values""Name of an existing Secret containing CLUSTER_TOKEN. Requires the external-secrets.io/type: webhook label when ESO is enabled
config.clusterTokentfy-agent Helm values""Inline cluster token. When set (and clusterTokenSecret is empty), the chart creates the token secret with the webhook label automatically
ENABLE_EXTERNAL_SECRETS_OPERATORControl plane environmentfalseEnables ESO-based manifest generation on the control plane
EXTERNAL_SECRETS_OPERATOR_REFRESH_INTERVALControl plane environment1hHow often ESO re-fetches secrets from the control plane
If you have questions or run into issues during the migration, reach out and we can help verify your tfy-agent and cluster token secret configuration.