> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# JVM metrics for Spark jobs

> See JVM heap, garbage collection, task and shuffle metrics for every Spark job run, and set them up for restricted and air-gapped clusters.

Every run of a Spark job shows JVM heap, garbage collection, task and shuffle metrics for its driver and executors in the run's **Metrics** tab, next to the [standard job metrics](https://www.truefoundry.com/docs/monitor-your-job.md#metrics-dashboard). They're on by default, so there's nothing to add to your job.

## Charts

| Chart | What it shows |
| - | - |
| JVM Heap Usage | Heap memory used across the driver and executors, and their combined maximum heap, in MB |
| GC Pause Time | Percentage of time each JVM spends paused in garbage collection. P50 is the median pod and P99 the pods with the most GC |
| GC Events | Garbage collections per minute across the driver and executors |
| Active Tasks | Tasks running across all executors |
| Completed Tasks | Tasks completed successfully across all executors |
| Shuffle Read/Write | Shuffle data read and written across all executors, in MB per second |

Each chart combines all pods of the run, so a run with 50 executors still shows one line per series.

## How it works

1. When a run starts, every driver and executor pod gets a small init container, `tfy-jmx-agent`. It downloads the [Prometheus JMX exporter](https://github.com/prometheus/jmx_exporter) Java agent (version 1.0.1) from Maven Central, checks its SHA-256 checksum, and saves it to a volume shared with the Spark container.
2. The Spark JVM loads the agent when it starts. The agent serves the JVM's and Spark's metrics on port `8090`.
3. Prometheus on the cluster scrapes the pods, and the **Metrics** tab reads the run's pods from it.

## Requirements

* **Network access to the jar.** Driver and executor pods download the jar from Maven Central (`https://repo1.maven.org/maven2`) by default. If your pods reach the internet through a proxy, set `HTTPS_PROXY`, `HTTP_PROXY` and `NO_PROXY`, in upper or lower case, as [environment variables](https://www.truefoundry.com/docs/environment-variables-and-secrets-jobs.md) on the Spark job. They can be plain values or secrets, and the download uses them. For clusters without internet access, see [Air-gapped clusters](#air-gapped-clusters).
* **Access to the downloader image.** Nodes must be able to pull `tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0`, which needs no credentials.
* **Port 8090 is reserved.** The agent listens on port `8090` in the driver and executor JVMs, so your job must not use it.
* **Spark metrics settings are managed.** TrueFoundry sets `spark.metrics.conf` and `spark.metrics.namespace` for the run, replacing any values in the job's Spark config. Java options in `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` are kept.

> [!WARNING]
>
> The init container has to finish before Spark starts. If the jar can't be downloaded or the downloader image can't be pulled, the run doesn't start. Fix the [settings](#settings), or [turn JVM metrics off](#turn-jvm-metrics-off).

## Settings

JVM metrics are configured on the control plane, as environment variables of `servicefoundryServer` in the values file of the `truefoundry` Helm chart. They apply to Spark jobs on every cluster connected to the control plane.

| Variable | Default | Purpose |
| - | - | - |
| `SPARK_JOB_JVM_METRICS_ENABLED` | `true` | Turns JVM metrics on or off for all Spark jobs |
| `SPARK_JMX_AGENT_MAVEN_REPOSITORY_URL` | `https://repo1.maven.org/maven2` | Repository the jar is downloaded from |
| `SPARK_JMX_AGENT_JAR_PATH` | `io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar` | Path of the jar in the repository. Set it only if the repository doesn't use the Maven layout |
| `SPARK_JMX_AGENT_DOWNLOADER_IMAGE` | `tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0` | Image the init container runs on. It needs `sh`, `sha256sum` and `curl`, or `wget` when the pods don't use a proxy |

### Turn JVM metrics off

```yaml truefoundry-values.yaml wrap lines theme={"dark"}
servicefoundryServer:
  env:
    SPARK_JOB_JVM_METRICS_ENABLED: "false"
```

Spark jobs then run without the agent, and the JVM metric charts are hidden.

## Air-gapped clusters

In an air-gapped cluster, pods can't reach Maven Central or `tfy.jfrog.io`. Make the jar and the downloader image available inside your network, then point the control plane at them. Do this before running Spark jobs on the cluster, or turn JVM metrics off until it's done.

**1. Add the jar to your artifact repository**

If your repository manager has a Maven repository that proxies Maven Central, use it as is and go to the next step.

Otherwise, download the jar on a machine with internet access and check it:

```bash wrap lines theme={"dark"}
curl -fLO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar
echo "7d61f737fd661610ccc14aea79764faa1ea94a340cbc8f0029b3d2edea3d80c1  jmx_prometheus_javaagent-1.0.1.jar" | sha256sum -c -
```

Upload it to a Maven repository at the standard path, `io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar`, or to any path in a generic repository.

The jar must be this exact file, because the init container rejects any other version by its checksum. The pods download it without credentials, so the repository must allow anonymous reads of the file.

**2. Copy the downloader image to your registry**

Copy the image with all its architectures, for example with `skopeo`:

```bash wrap lines theme={"dark"}
skopeo copy --all docker://tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0 docker://<registry_url>/tfy-mirror/curlimages/curl:8.23.0
```

The image isn't listed in the `artifacts-manifest.json` that the upload script reads, so copy it separately. If you set up the Kyverno registry rewrite from the [compute plane air-gapped guide](https://www.truefoundry.com/docs/infrastructure/deploy-compute-plane.md#how-to-install-in-an-air-gapped--restricted-network-environment), keep this path: Kyverno rewrites the init container's image to your registry, and you don't need to set `SPARK_JMX_AGENT_DOWNLOADER_IMAGE`.

Nodes must be able to pull the image without credentials, with registry credentials configured on the nodes, or from the same registry and credentials as your Spark image.

**3. Point the control plane at your repository**

```yaml truefoundry-values.yaml wrap lines theme={"dark"}
servicefoundryServer:
  env:
    SPARK_JMX_AGENT_MAVEN_REPOSITORY_URL: "https://artifactory.example.com/artifactory/maven-remote"
    # Only if the jar isn't at the Maven path in the repository
    # SPARK_JMX_AGENT_JAR_PATH: "tools/jmx/jmx_prometheus_javaagent-1.0.1.jar"
    # Only if Kyverno doesn't rewrite the image to your registry
    # SPARK_JMX_AGENT_DOWNLOADER_IMAGE: "<registry_url>/tfy-mirror/curlimages/curl:8.23.0"
```

Then upgrade the Helm release as shown in [Settings](#settings). The settings apply to every cluster connected to the control plane, so each of those clusters must be able to reach the repository.

**4. Check a run**

Trigger a run of a Spark job. The log of the driver pod's `tfy-jmx-agent` init container should end with:

```text theme={"dark"}
/tfy-jmx-agent/jmx_prometheus_javaagent.jar: OK
```

The charts appear in the run's **Metrics** tab within a couple of minutes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.