Container Images

The container image system in this SDK provides a fluent, high-level API for defining and building environment specifications. Instead of writing Dockerfiles manually, you use the Image class to programmatically define base images and additional layers such as system packages, Python dependencies, and source code.

The Fluent Image API

The Image class (found in src/flyte/_image.py) follows a fluent, two-step pattern:

1. Initialization: Create a base image using a from_* constructor.
2. Customization: Add layers using with_* methods.

A critical design aspect of the Image class is immutability. Every with_* method calls the internal clone() method, returning a new Image instance with the additional layer. This allows you to reuse base image definitions without side effects.

Direct instantiation of Image is disallowed via a guard in __post_init__. You must use the provided class-level constructors.

from flyte import Image

# Defining a reusable base
base = Image.from_debian_base(python_version=(3, 12))

# Creating specialized images from the base
training_image = base.with_pip_packages("torch", "numpy")
serving_image = base.with_pip_packages("fastapi", "uvicorn")

Base Image Constructors

The SDK provides several ways to start an image definition depending on your requirements:

Debian Base

Image.from_debian_base() is the standard way to start. It uses a slim Debian-based image optimized for Flyte. It automatically detects the local Python version if not specified and can optionally install the Flyte library itself.

Custom Base URI

If you have an existing image in a registry, use Image.from_base(image_uri="..."). This allows you to layer Flyte-specific configurations on top of corporate or specialized base images.

Dockerfile Support

For complex setups that cannot be expressed through the fluent API, Image.from_dockerfile() allows you to point to an existing Dockerfile.

• Note: Images created from a Dockerfile are marked as extendable=False. You cannot add further layers using with_* methods to a Dockerfile-based image because the SDK does not parse the Dockerfile to understand its internal state (like the OS or Python environment).

UV Scripts

Image.from_uv_script() is a specialized constructor that creates an image from a uv-compatible script. It parses the script's header for inline dependencies and Python version requirements, making it ideal for self-contained scripts.

# Example from examples/image/uv_image.py
image = Image.from_uv_script(__file__, name="hello", registry="ghcr.io/flyteorg")

Layering and Customization

Once a base is established, you can add various layers. Each layer is encapsulated in a specific class (e.g., PipPackages, AptPackages, Env) and added to the _layers tuple of the Image object.

Dependency Management

• with_pip_packages(*packages): Adds a layer that runs pip install.
• with_apt_packages(*packages): Adds a layer for system-level dependencies via apt-get.
• with_requirements(file): Installs dependencies from a standard requirements.txt.

Environment and Commands

• with_env_vars(env_vars): Sets environment variables in the container.
• with_commands(commands): Runs arbitrary shell commands during the build process.
• with_workdir(workdir): Sets the WORKDIR for subsequent layers and the runtime.

Source Code Integration

The SDK provides multiple ways to include your code:

• with_source_folder(src, dst): Copies a local directory into the image.
• with_source_file(src, dst): Copies specific files.
• with_code_bundle(): A specialized layer that automatically handles copying source code based on the runner's configuration.

UV and Poetry Integration

The SDK treats uv and poetry as first-class citizens for dependency management.

• UV Projects: with_uv_project() looks for pyproject.toml and uv.lock. It can be configured to install only dependencies or the entire project as a package using the project_install_mode parameter.
• Poetry Projects: with_poetry_project() provides similar functionality for Poetry-managed environments, handling poetry.lock and project installation.

Building Images

The ImageBuildEngine (in src/flyte/_internal/imagebuild/image_builder.py) is responsible for transforming an Image specification into a real container image.

The Build Process

When you call flyte.build(image), the engine performs the following steps:

1. Validation: Calls image.validate() to ensure layers are consistent.
2. Existence Check: It calculates a hash of the image (based on base image and all layers). If an image with that hash already exists in the registry, it skips the build unless force=True or the tag is latest.
3. Builder Selection: It selects a builder based on configuration (FLYTE_IMAGE_BUILDER).
   • Local Builder: Uses DockerImageBuilder, which requires docker buildx. It creates a builder instance named flytex to handle multi-platform builds.
   • Remote Builder: Uses RemoteImageBuilder to offload the build process to a remote service.

Build Results

The build() function returns an ImageBuild object:

@dataclass
class ImageBuild:
    uri: str | None            # The final URI (e.g., registry/name:hash)
    remote_run: Optional[Run]  # The Run object if a remote builder was used

Hashing and Caching

To ensure efficient builds, the SDK implements a robust hashing mechanism in Image._get_hash_digest(). The hash is derived from:

• The base_image URI.
• The contents of the dockerfile (if applicable).
• Every layer in the _layers tuple.

Each layer class implements update_hash(hasher, ignore). For example, a PipPackages layer hashes the list of packages and installation arguments, while a CopyConfig layer (used by with_source_folder) hashes the actual contents of the files being copied, respecting .dockerignore files.

If the resulting hash matches an image already present in the target registry, the SDK avoids a redundant build, significantly speeding up deployment cycles.

Secret Mounts

For security-sensitive builds (e.g., installing from private PyPI indexes or Git repositories), many with_* methods support secret_mounts. These allow you to safely pass credentials to the build engine without baking them into the image layers.

# Example of using secrets for private packages
image = (
    Image.from_debian_base()
    .with_pip_packages(
        "private-pkg", 
        secret_mounts=[Secret(key="PYPI_TOKEN")]
    )
)

During the build, these secrets are mounted as environment variables or files, depending on the builder's implementation.