Task Config

​

Things you can define in task config

• env: you can pass the environment variables as env in task config, where env is a dictionary of key-value pairs.
• service_account: you can pass the service_account name in the task config which is necessary to save the input and output data of the task.
• resources: You can define the resource to allocate to each of the tasks, where you can define the cpu limit, storage limit, memory limit, GPU types, etc. You can refer to this article for more information on each.
• mounts: You can attach volume mounts such as volume mounts, string mounts or secret mounts. You can learn more about mounts and how to use them in workflow in this guide.

​

Types of task config

• There are two types of task config PythonTaskConfig and ContainerTaskConfig.
  • PythonTaskConfig: This task config can be passed in the normal python task in the task decorator. You can define the environment variables, Resources, service account, and the image spec in PythonTaskConfig. The image spec can be of two types TaskPythonBuild and TaskDockerFileBuild.
    • TaskPythonBuild is used when you do not have a Dockerfile and you want to build an image where you want to specify the pip packages, apt packages or requirements file path in the build spec, then TaskPythonBuild is used.
    • TaskDockerFileBuild is used when you already have a Dockerfile and you just want to build then you use TaskDockerFileBuild.
  • ContainerTaskConfig: This task config can be used when you already have a docker image and you want to use that as a task in the workflow directly or you have code uploaded on GitHub or the remote source. There you have a docker file which you want to use as a task in the workflow.
  • PySparkTaskConfig: This task config is used for Spark tasks that run distributed PySpark jobs. The image spec can be of two types:
    • TaskPySparkBuild is used when you want TrueFoundry to build a Spark image with your code and dependencies. You can specify the Spark version, pip packages, apt packages, or requirements file path.
    • TaskSparkImage is used when you already have a pre-built Spark image that contains all your workflow code and dependencies. This skips the Docker build phase entirely, making deployments faster. Your image must contain:
      1. All workflow source code at /app (or appropriate PYTHONPATH)
      2. truefoundry[workflow,spark] package installed
      3. PySpark version matching the spark_version parameter
      4. Hadoop AWS/GCS/Azure JARs if using cloud storage

​

Building a TaskSparkImage-Compatible Image

1. The Spark driver pod imports your task function - Without the code, Python cannot import the module
2. Flytekit deserializes inputs and calls your function - The workflow runtime needs to be installed
3. No code injection happens at deploy time - Unlike TaskPySparkBuild, the image is used as-is

​

Required Directory Structure

/opt/venv/bin/
    python                    # Python interpreter (required path)
    entrypoint.py             # Flytekit entrypoint (auto-installed with flytekit)
/opt/venv/lib/.../site-packages/
    flytekit/                 # Flytekit for task execution
    truefoundry/              # TrueFoundry workflow runtime
    pyspark/                  # PySpark library
$SPARK_HOME/jars/
    hadoop-aws-3.3.4.jar      # Required for S3 access
    aws-java-sdk-bundle-1.12.262.jar
/app/
    workflow.py               # Your workflow code (must be importable)
    # ... other source files

​

Example Dockerfile

FROM apache/spark:3.5.2-scala2.12-java17-python3-ubuntu

USER root

# Install dependencies
RUN apt-get update && apt-get install -y git python3-venv curl && rm -rf /var/lib/apt/lists/*

# Install uv for faster package installation (optional, can use pip instead)
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
ENV PATH="/root/.local/bin:$PATH"

# Install Hadoop AWS JARs for S3 access (required for cloud storage)
RUN curl -fSL -o $SPARK_HOME/jars/hadoop-aws-3.3.4.jar \
    https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar && \
    curl -fSL -o $SPARK_HOME/jars/aws-java-sdk-bundle-1.12.262.jar \
    https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar

# Create Python virtual environment at the required path
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install required dependencies into the venv
RUN uv pip install --python /opt/venv/bin/python --no-cache \
    truefoundry[workflow,spark] \
    pyspark==3.5.2

# Copy your workflow source code
COPY . /app
WORKDIR /app
ENV PYTHONPATH=/app

docker build --platform linux/amd64 -t my-spark-image:v1.0 .

from truefoundry.deploy import Image, NvidiaGPU, Resources
from truefoundry.workflow import (
    ContainerTask,
    ContainerTaskConfig,
    ExecutionConfig,
    FlyteDirectory,
    PySparkTaskConfig,
    PythonTaskConfig,
    TaskPySparkBuild,
    TaskPythonBuild,
    TaskSparkImage,
    conditional,
    map_task,
    task,
    workflow,
)
from truefoundry.deploy.v2.lib.patched_models import (
    SparkDriverConfig,
    SparkExecutorConfig,
    SparkExecutorFixedInstances,
)

# Python task config example
task_config = PythonTaskConfig(
    image=TaskPythonBuild(
        python_version="3.9",
        pip_packages=["truefoundry[workflow]"],
    ),
    resources=Resources(cpu_request=0.5, cpu_limit=0.5),
    service_account="<service-account>",
)

# Container task config example
echo = ContainerTask(
    name="echo",
    task_config=ContainerTaskConfig(
        image=Image(
            image_uri="bash:4.1",
            command=["echo", "hello"],
        ),
        service_account="<service-account>",
    ),
)

# PySpark task config with TaskPySparkBuild (builds image)
spark_build_config = PySparkTaskConfig(
    image=TaskPySparkBuild(
        spark_version="3.5.2",
        pip_packages=["truefoundry[workflow,spark]"],
    ),
    service_account="spark-service-account",
    driver_config=SparkDriverConfig(
        resources=Resources(cpu_request=1, cpu_limit=1, memory_request=1024, memory_limit=1024),
    ),
    executor_config=SparkExecutorConfig(
        instances=SparkExecutorFixedInstances(count=2),
        resources=Resources(cpu_request=1, cpu_limit=1, memory_request=1024, memory_limit=1024),
    ),
)

# PySpark task config with TaskSparkImage (skips build, uses pre-built image)
spark_image_config = PySparkTaskConfig(
    image=TaskSparkImage(
        spark_version="3.5.2",
        container_image="my-registry.com/my-spark-image:v1.0",
        docker_registry="my-docker-registry-fqn",  # optional, for private registries
    ),
    service_account="spark-service-account",
    driver_config=SparkDriverConfig(
        resources=Resources(cpu_request=1, cpu_limit=1, memory_request=1024, memory_limit=1024),
    ),
    executor_config=SparkExecutorConfig(
        instances=SparkExecutorFixedInstances(count=2),
        resources=Resources(cpu_request=1, cpu_limit=1, memory_request=1024, memory_limit=1024),
    ),
)

...