JVM metrics for Spark jobs

Every run of a Spark job shows JVM heap, garbage collection, task and shuffle metrics for its driver and executors in the run’s Metrics tab, next to the standard job metrics. They’re on by default, so there’s nothing to add to your job.

Charts

Chart	What it shows
JVM Heap Usage	Heap memory used across the driver and executors, and their combined maximum heap, in MB
GC Pause Time	Percentage of time each JVM spends paused in garbage collection. P50 is the median pod and P99 the pods with the most GC
GC Events	Garbage collections per minute across the driver and executors
Active Tasks	Tasks running across all executors
Completed Tasks	Tasks completed successfully across all executors
Shuffle Read/Write	Shuffle data read and written across all executors, in MB per second

Each chart combines all pods of the run, so a run with 50 executors still shows one line per series.

How it works

When a run starts, every driver and executor pod gets a small init container, tfy-jmx-agent. It downloads the Prometheus JMX exporter Java agent (version 1.0.1) from Maven Central, checks its SHA-256 checksum, and saves it to a volume shared with the Spark container.
The Spark JVM loads the agent when it starts. The agent serves the JVM’s and Spark’s metrics on port 8090.
Prometheus on the cluster scrapes the pods, and the Metrics tab reads the run’s pods from it.

Requirements

Network access to the jar. Driver and executor pods download the jar from Maven Central (https://repo1.maven.org/maven2) by default. If your pods reach the internet through a proxy, set HTTPS_PROXY, HTTP_PROXY and NO_PROXY, in upper or lower case, as environment variables on the Spark job. They can be plain values or secrets, and the download uses them. For clusters without internet access, see Air-gapped clusters.
Access to the downloader image. Nodes must be able to pull tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0, which needs no credentials.
Port 8090 is reserved. The agent listens on port 8090 in the driver and executor JVMs, so your job must not use it.
Spark metrics settings are managed. TrueFoundry sets spark.metrics.conf and spark.metrics.namespace for the run, replacing any values in the job’s Spark config. Java options in spark.driver.extraJavaOptions and spark.executor.extraJavaOptions are kept.

The init container has to finish before Spark starts. If the jar can’t be downloaded or the downloader image can’t be pulled, the run doesn’t start. Fix the settings, or turn JVM metrics off.

Settings

JVM metrics are configured on the control plane, as environment variables of servicefoundryServer in the values file of the truefoundry Helm chart. They apply to Spark jobs on every cluster connected to the control plane.

Variable	Default	Purpose
`SPARK_JOB_JVM_METRICS_ENABLED`	`true`	Turns JVM metrics on or off for all Spark jobs
`SPARK_JMX_AGENT_MAVEN_REPOSITORY_URL`	`https://repo1.maven.org/maven2`	Repository the jar is downloaded from
`SPARK_JMX_AGENT_JAR_PATH`	`io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar`	Path of the jar in the repository. Set it only if the repository doesn’t use the Maven layout
`SPARK_JMX_AGENT_DOWNLOADER_IMAGE`	`tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0`	Image the init container runs on. It needs `sh`, `sha256sum` and `curl`, or `wget` when the pods don’t use a proxy

Turn JVM metrics off

truefoundry-values.yaml

servicefoundryServer:
  env:
    SPARK_JOB_JVM_METRICS_ENABLED: "false"

Spark jobs then run without the agent, and the JVM metric charts are hidden.

Air-gapped clusters

In an air-gapped cluster, pods can’t reach Maven Central or tfy.jfrog.io. Make the jar and the downloader image available inside your network, then point the control plane at them. Do this before running Spark jobs on the cluster, or turn JVM metrics off until it’s done.

Add the jar to your artifact repository

If your repository manager has a Maven repository that proxies Maven Central, use it as is and go to the next step.Otherwise, download the jar on a machine with internet access and check it:

curl -fLO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar
echo "7d61f737fd661610ccc14aea79764faa1ea94a340cbc8f0029b3d2edea3d80c1  jmx_prometheus_javaagent-1.0.1.jar" | sha256sum -c -

Upload it to a Maven repository at the standard path, io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar, or to any path in a generic repository.The jar must be this exact file, because the init container rejects any other version by its checksum. The pods download it without credentials, so the repository must allow anonymous reads of the file.

Copy the downloader image to your registry

Copy the image with all its architectures, for example with skopeo:

skopeo copy --all docker://tfy.jfrog.io/tfy-mirror/curlimages/curl:8.23.0 docker://<registry_url>/tfy-mirror/curlimages/curl:8.23.0

The image isn’t listed in the artifacts-manifest.json that the upload script reads, so copy it separately. If you set up the Kyverno registry rewrite from the compute plane air-gapped guide, keep this path: Kyverno rewrites the init container’s image to your registry, and you don’t need to set SPARK_JMX_AGENT_DOWNLOADER_IMAGE.Nodes must be able to pull the image without credentials, with registry credentials configured on the nodes, or from the same registry and credentials as your Spark image.

Point the control plane at your repository

truefoundry-values.yaml

servicefoundryServer:
  env:
    SPARK_JMX_AGENT_MAVEN_REPOSITORY_URL: "https://artifactory.example.com/artifactory/maven-remote"
    # Only if the jar isn't at the Maven path in the repository
    # SPARK_JMX_AGENT_JAR_PATH: "tools/jmx/jmx_prometheus_javaagent-1.0.1.jar"
    # Only if Kyverno doesn't rewrite the image to your registry
    # SPARK_JMX_AGENT_DOWNLOADER_IMAGE: "<registry_url>/tfy-mirror/curlimages/curl:8.23.0"

Then upgrade the Helm release as shown in Settings. The settings apply to every cluster connected to the control plane, so each of those clusters must be able to reach the repository.

Check a run

Trigger a run of a Spark job. The log of the driver pod’s tfy-jmx-agent init container should end with:

/tfy-jmx-agent/jmx_prometheus_javaagent.jar: OK

The charts appear in the run’s Metrics tab within a couple of minutes.

Introduction

Train and Deploy Models

Service Deployment

Job Deployment

LLM Deployment

LLM Finetuning

MCP Server Deployment

Workflow Deployment

Async Service Deployment

Volumes

Experiment Tracking

Advanced Features

Charts

How it works

Requirements

Settings

Turn JVM metrics off

Air-gapped clusters

​Charts

​How it works

​Requirements

​Settings

​Turn JVM metrics off

​Air-gapped clusters

Charts

How it works

Requirements

Settings

Turn JVM metrics off

Air-gapped clusters