> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom CA Certificate Injection
> Configure TrueFoundry to trust TLS certificates issued by an internal or private Certificate Authority across the control plane, compute plane, and workloads.
If your organization signs TLS certificates with a private Certificate Authority (CA) instead of a public one, clients do not trust those certificates by default. TLS handshakes fail with errors such as `x509: certificate signed by unknown authority` or `unable to verify the first certificate`.
To fix this, the CA bundle must be present in the trust store of every pod that opens the connection. TrueFoundry supports this in three scenarios.
| Scenario | Who needs the CA | How to configure it |
| ------------------------------------------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------- |
| Control plane calls an endpoint served with a custom CA certificate | Control plane pods such as `servicefoundry-server` and the AI Gateway | `global.customCA` in the `truefoundry` chart |
| Compute plane connects to the control plane | `tfy-agent` pods | `global.customCA` in the `tfy-agent` chart |
| Workloads call an endpoint served with a custom CA certificate | Any pod in a workspace namespace | Kyverno policies from the `tfy-kyverno-config` chart |
This configuration is only needed for self-signed or internal CA certificates. Endpoints with certificates issued by a public CA are already trusted.
## Control plane calling endpoints signed by a custom CA
Control plane components do not talk to each other over TLS — traffic between them is plain pod-to-pod communication inside the cluster. The control plane needs a custom CA when TrueFoundry services make outbound calls to endpoints that present a certificate signed by your private CA, such as model endpoints, integration provider endpoints, private registries, or corporate proxies.
Set the CA in your `truefoundry` chart values:
```yaml truefoundry-values.yaml wrap lines theme={"dark"}
global:
customCA:
# Enable custom CA certificate injection via initContainers
enabled: true
# PEM-encoded CA certificate. An initContainer merges it with the system CAs.
certificate: |
-----BEGIN CERTIFICATE-----
... (your CA certificate) ...
-----END CERTIFICATE-----
```
You can also reference an existing ConfigMap that holds the certificate under the key `ca-certificates.crt`:
```yaml truefoundry-values.yaml wrap lines theme={"dark"}
global:
customCA:
enabled: true
existingConfigMap:
name: custom-ca-certificates
# When false, an initContainer merges your CA with the system CA bundle.
# When true, the ConfigMap is mounted directly at /etc/ssl/certs/ and
# replaces the system bundle, so it must contain system CAs + your CAs.
overrideCAList: false
```
For the full walkthrough, including how to build the ConfigMap and upgrade the release, see [How to configure custom CA certificates](/docs/platform/deploy-control-plane-faq#how-to-configure-custom-ca-certificates).
## Compute plane connecting to the control plane
`tfy-agent` runs on the compute plane and connects to your control plane URL. If that URL serves a certificate signed by a private CA, or the traffic passes through a corporate TLS proxy, the agent needs the CA bundle.
Set the CA in your `tfy-agent` chart values. When enabled, the bundle is mounted into the `tfyAgent`, `tfyAgentProxy`, and `sdsServer` pods:
```yaml tfy-agent-values.yaml wrap lines theme={"dark"}
global:
customCA:
enabled: true
certificate: |
-----BEGIN CERTIFICATE-----
... (your CA certificate) ...
-----END CERTIFICATE-----
# Or reference an existing ConfigMap with the key "ca-certificates.crt"
# existingConfigMap:
# name: custom-ca-certificates
# overrideCAList: false
```
Install or upgrade `tfy-agent` after the control plane is reachable, so the agent can complete its first connection. See [TFY Agent](/docs/tfy-agent) for the full agent configuration.
## Workloads calling endpoints signed by a custom CA
Workloads deployed through TrueFoundry — services, jobs, notebooks, and LLM deployments — run from arbitrary container images in workspace namespaces. When these workloads call an endpoint that presents a certificate signed by your private CA, they need the CA bundle too, and no single Helm value covers images you do not build.
TrueFoundry handles this with two Kyverno policies packaged in the [`tfy-kyverno-config`](https://github.com/truefoundry/infra-charts/tree/main/charts/tfy-kyverno-config) chart:
| Policy | What it does |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| ConfigMap sync | Clones the CA bundle ConfigMap from a source namespace such as `truefoundry` into every matching workspace namespace and keeps the copies in sync |
| Pod volume mounts | Mutates pods at admission to mount the CA bundle at `/etc/ssl/certs/ca-certificates.crt` and set the SSL environment variables |
Together they remove the need to rebuild images, patch pod specs, or distribute certificates by hand.
### Prerequisites
* A compute plane cluster connected to your TrueFoundry control plane.
* Cluster admin access, because both policies are cluster-scoped resources.
* Kyverno installed on the cluster.
* `tfy-kyverno-config` chart version `0.1.10` or later.
* Your internal CA certificate in PEM format, referred to below as `custom-ca.crt`.
The bundle you distribute must contain the public CA certificates and your internal CA. If you ship only your internal CA, workloads lose trust for every public endpoint.
Extract a current public bundle from a standard image:
```bash wrap lines theme={"dark"}
docker run --rm nginx:latest cat /etc/ssl/certs/ca-certificates.crt > public-ca-bundle.crt
```
Check that your internal CA certificate is valid:
```bash wrap lines theme={"dark"}
openssl x509 -in custom-ca.crt -noout -text
```
Append it to the public bundle:
```bash wrap lines theme={"dark"}
cat public-ca-bundle.crt custom-ca.crt > ca-certificates.crt
```
Create the ConfigMap in a source namespace. The examples use the `truefoundry` namespace and the ConfigMap name `ca-cert-bundle`, which is the chart default. The key must be `ca-certificates.crt`.
```bash wrap lines theme={"dark"}
kubectl create configmap ca-cert-bundle \
--from-file=ca-certificates.crt=ca-certificates.crt \
-n truefoundry
```
To update an existing ConfigMap, apply it instead:
```bash wrap lines theme={"dark"}
kubectl create configmap ca-cert-bundle \
--from-file=ca-certificates.crt=ca-certificates.crt \
-n truefoundry --dry-run=client -o yaml | kubectl apply -f -
```
The sync policy watches this ConfigMap, so future certificate rotations only need this one update.
```bash wrap lines theme={"dark"}
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update kyverno
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace
```
If Kyverno is already installed on the cluster, skip this step and reuse the existing installation.
Create the chart values:
```yaml kyverno-config-values.yaml wrap lines theme={"dark"}
syncConfigMaps:
enabled: true
includeNamespaces:
- "tfy-*"
items:
- namespace: truefoundry
name: ca-cert-bundle
podVolumeMounts:
enabled: true
includeNamespaces:
- "tfy-*"
configMapName: ca-cert-bundle
mountInitContainers: true
```
`tfy-*` matches the namespaces that TrueFoundry creates for workspaces. Adjust the include list if your workspaces use a different naming pattern, and add `excludeNamespaces` for namespaces that must be skipped.
Deploying from the platform records the chart version and change history, which makes later upgrades easier to track.
1. In the TrueFoundry dashboard, go to **Deployments** and click **New**.
2. Click **Show advanced** and select **Helm**.
3. Choose **Public Helm Repository** as the chart source and fill in:
* **Helm repository URL**: `https://truefoundry.github.io/infra-charts`
* **Chart name**: `tfy-kyverno-config`
* **Version**: the chart version you want to pin
4. Paste your values into the values editor and click **Submit**.
See [Deploy Helm Charts](/docs/deploy-helm-charts) for the full flow, including OCI registry and Git repository sources for air-gapped clusters.
These policies are cluster-scoped. Deploying Helm charts that create cluster-scoped objects requires cluster admin privileges. If your setup does not allow that, install the chart with the Helm CLI instead.
```bash wrap lines theme={"dark"}
helm repo add truefoundry https://truefoundry.github.io/infra-charts
helm repo update truefoundry
helm install tfy-kyverno-config truefoundry/tfy-kyverno-config \
-n kyverno --create-namespace -f kyverno-config-values.yaml
```
Confirm both policies exist. You should see `-sync-configmaps` and `-pod-volume-mounts`:
```bash wrap lines theme={"dark"}
kubectl get clusterpolicy
```
Confirm the ConfigMap was cloned into a workspace namespace:
```bash wrap lines theme={"dark"}
kubectl get configmap ca-cert-bundle -n tfy-myworkspace
```
Deploy or restart a workload, then check the mount and the environment variables:
```bash wrap lines theme={"dark"}
kubectl exec -n tfy-myworkspace -- ls -l /etc/ssl/certs/ca-certificates.crt
kubectl exec -n tfy-myworkspace -- env | grep -E 'SSL_CERT_FILE|REQUESTS_CA_BUNDLE|SSL_CERT_DIR'
```
Confirm the workload reaches an endpoint that uses your internal CA:
```bash wrap lines theme={"dark"}
kubectl exec -n tfy-myworkspace -- curl -sS -o /dev/null -w '%{http_code}\n' https://internal.example.com
```
### Policies created by the chart
The chart renders these manifests from your values. They are shown here so you can review what is applied to the cluster.
ConfigMap sync:
```yaml wrap lines theme={"dark"}
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tfy-kyverno-config-sync-configmaps
spec:
rules:
- name: sync-configmap-ca-cert-bundle
skipBackgroundRequests: false
match:
any:
- resources:
kinds:
- Namespace
names:
- "tfy-*"
generate:
generateExisting: true
synchronize: true
apiVersion: v1
kind: ConfigMap
name: ca-cert-bundle
namespace: "{{request.object.metadata.name}}"
clone:
namespace: truefoundry
name: ca-cert-bundle
```
Pod volume mounts. The init container rule mirrors the container rule and is rendered only when `mountInitContainers` is enabled:
```yaml wrap lines theme={"dark"}
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tfy-kyverno-config-pod-volume-mounts
spec:
background: false
rules:
- name: add-volumes
match:
resources:
kinds:
- Pod
namespaces:
- "tfy-*"
mutate:
patchStrategicMerge:
spec:
volumes:
- name: custom-ca-cert
configMap:
name: ca-cert-bundle
items:
- key: ca-certificates.crt
path: ca-certificates.crt
- name: patch-all-containers
match:
resources:
kinds:
- Pod
namespaces:
- "tfy-*"
mutate:
foreach:
- list: request.object.spec.containers
patchStrategicMerge:
spec:
containers:
- name: "{{ element.name }}"
env:
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca-certificates.crt
- name: REQUESTS_CA_BUNDLE
value: /etc/ssl/certs/ca-certificates.crt
- name: SSL_CERT_DIR
value: /etc/ssl/certs
volumeMounts:
- name: custom-ca-cert
mountPath: /etc/ssl/certs/ca-certificates.crt
subPath: ca-certificates.crt
readOnly: true
```
### Configuration reference
| Parameter | Description | Default |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------- |
| `syncConfigMaps.enabled` | Enable ConfigMap syncing across namespaces | `false` |
| `syncConfigMaps.useClusterPolicy` | Render a `ClusterPolicy` (`kyverno.io/v1`) instead of a `GeneratingPolicy`. Required for wildcard namespace patterns | `true` |
| `syncConfigMaps.includeNamespaces` | Namespaces to sync into. When non-empty, only these are matched. Supports wildcards such as `tfy-*` | `[]` |
| `syncConfigMaps.excludeNamespaces` | Namespaces to skip | `[]` |
| `syncConfigMaps.items` | ConfigMaps to sync. Each entry takes `namespace` (the source) and `name` | `[]` |
The generated policy sets `generateExisting: true` and `synchronize: true`, so it populates namespaces that already exist as well as new ones, and propagates later changes to the source ConfigMap.
| Parameter | Description | Default |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| `podVolumeMounts.enabled` | Enable volume injection into pods | `false` |
| `podVolumeMounts.useClusterPolicy` | Render a `ClusterPolicy` (`kyverno.io/v1`) instead of a `MutatingPolicy`. Required for wildcard namespaces, environment variables, `readOnly` mounts, and ConfigMap key selection | `true` |
| `podVolumeMounts.policyName` | Name of the generated `ClusterPolicy` | `-pod-volume-mounts` |
| `podVolumeMounts.configMapName` | ConfigMap to mount for the built-in CA mount | `ca-cert-bundle` |
| `podVolumeMounts.mountInitContainers` | Also mount the volume into init containers | `false` |
| `podVolumeMounts.includeNamespaces` | Namespaces to mutate. When non-empty, only these are matched. Supports wildcards such as `tfy-*` | `[]` |
| `podVolumeMounts.excludeNamespaces` | Namespaces to skip | `[]` |
| `podVolumeMounts.objectSelector` | Label selector that restricts which pods are mutated | `{}` |
| `podVolumeMounts.mountDetails` | Replaces the chart's default mount list | `[]` |
| `podVolumeMounts.additionalMountDetails` | Appends extra mounts to the default list | `[]` |
Each entry in `mountDetails` and `additionalMountDetails` accepts:
| Field | Required | Description |
| ---------------- | -------------- | ------------------------------------------------------- |
| `type` | Yes | `configMap` or `secret` |
| `configMapName` | For ConfigMaps | Name of the ConfigMap to mount |
| `secretName` | For Secrets | Name of the Secret to mount |
| `volumeName` | No | Volume name. Defaults to the ConfigMap or Secret name |
| `mountPath` | Yes | Path inside the container |
| `subPath` | No | Sub-path within the volume |
| `readOnly` | No | Mount read-only |
| `configMapItems` | No | Key-to-path mapping for ConfigMap volumes |
| `envVars` | No | Environment variables to set on every matched container |
For example, to mount a TLS Secret alongside the default CA bundle:
```yaml kyverno-config-values.yaml wrap lines theme={"dark"}
podVolumeMounts:
enabled: true
includeNamespaces:
- "tfy-*"
additionalMountDetails:
- type: secret
secretName: internal-tls
mountPath: /etc/tls
readOnly: true
```
When `mountDetails` is empty, the chart mounts the CA bundle and sets the variables that most runtimes read:
| Setting | Value |
| -------------------- | ----------------------------------------------------------------------------------- |
| Mount path | `/etc/ssl/certs/ca-certificates.crt` with `subPath: ca-certificates.crt`, read-only |
| `SSL_CERT_FILE` | `/etc/ssl/certs/ca-certificates.crt` |
| `REQUESTS_CA_BUNDLE` | `/etc/ssl/certs/ca-certificates.crt` |
| `SSL_CERT_DIR` | `/etc/ssl/certs` |
Setting `mountDetails` replaces these defaults, including the environment variables. To keep the CA mount and add more volumes, use `additionalMountDetails`.
Some runtimes keep their own trust store and ignore these variables. For Node.js applications, add `NODE_EXTRA_CA_CERTS` pointing at the same path. For Java applications, import the CA into the JVM truststore.
### Troubleshooting
| Symptom | Likely cause | Fix |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| ConfigMap missing in a workspace namespace | Namespace does not match `includeNamespaces`, or wildcards are used with `useClusterPolicy: false` | Correct the include list and keep `useClusterPolicy: true`, then check `kubectl describe clusterpolicy -sync-configmaps` |
| Pods do not have the volume | Pods were created before the policy was installed | Restart the workload; the policy mutates pods at admission |
| Volume present but TLS still fails | Bundle is missing the internal CA, or the endpoint serves an incomplete chain | Verify with `openssl s_client -connect internal.example.com:443 -CAfile /etc/ssl/certs/ca-certificates.crt` inside the pod |
| Calls to public endpoints break | Bundle contains only the internal CA | Rebuild the combined bundle and update the source ConfigMap; the sync policy propagates the fix |
| Application ignores the mounted bundle | Runtime uses its own trust store | Set `NODE_EXTRA_CA_CERTS` for Node.js, or import the CA into the JVM truststore for Java |
Use these commands to narrow down where the injection stopped.
Check whether the sync policy ran and what it reported:
```bash wrap lines theme={"dark"}
kubectl describe clusterpolicy -sync-configmaps
```
Check whether a running pod received the volume:
```bash wrap lines theme={"dark"}
kubectl get pod -n tfy-myworkspace -o yaml | grep -A 5 'custom-ca-cert'
```
Check the certificate chain that the endpoint serves against the mounted bundle:
```bash wrap lines theme={"dark"}
kubectl exec -n tfy-myworkspace -- \
openssl s_client -connect internal.example.com:443 -CAfile /etc/ssl/certs/ca-certificates.crt