Skip to main content

Git Integration and Source Linking

Flyte integrates with Git to provide deep traceability between deployed tasks and their source code. By capturing the state of the local repository during deployment, Flyte can generate direct links to the specific file and line number where a task is defined on platforms like GitHub and GitLab.

Git Repository Discovery​

The core of this integration is the GitStatus class in flyte.git._config. It acts as a container for the current state of a Git repository, including the remote URL, the current commit SHA, and the status of the working tree.

The GitStatus.from_current_repo() method automatically discovers this information by executing Git commands via the subprocess module:

  • Root Directory: Discovered using git rev-parse --show-toplevel.
  • Commit SHA: Retrieved using git rev-parse HEAD.
  • Tree Status: Checked via git status --porcelain. If the output is empty, the tree is considered "clean."
  • Remote URL: Prefers the origin push URL, falling back to the first available remote alphabetically.

URL Normalization​

To ensure that source links are accessible via a web browser, GitStatus normalizes remote URLs to HTTPS format. The _normalize_url_to_https method handles conversions such as:

  • git@github.com:user/repo.git → https://github.com/user/repo
  • https://github.com/user/repo.git → https://github.com/user/repo

Source Code Linking​

When a task is deployed, Flyte uses GitStatus to build a source code URL. This URL is then embedded in the task's metadata (specifically the DocumentationEntity within the task definition).

URL Builders​

The logic for constructing platform-specific URLs is encapsulated in classes implementing the GitUrlBuilder protocol. The GIT_URL_BUILDER_REGISTRY maps hostnames to these builders:

  • GitHub: GithubUrlBuilder generates URLs in the format {remote}/blob/{sha}/{path}.
  • GitLab: GitlabUrlBuilder generates URLs in the format {remote}/-/blob/{sha}/{path}.

The "Clean Tree" Requirement​

A critical safety feature in GitStatus.build_url is the handling of local changes. Line numbers (e.g., #L123) are only appended to the generated URL if the working tree is clean (is_tree_clean=True). This prevents the UI from linking to an incorrect line number if the local file has been modified since the last commit.

Deployment Integration​

In src/flyte/_deploy.py, Flyte automatically extracts the function's line number and filename to generate these links:

from flyte.git import GitStatus

# Inside deployment logic
line_number = task_template.func.__code__.co_firstlineno + 1
file_path = task_template.func.__code__.co_filename

git_status = GitStatus.from_current_repo()
if git_status.is_valid:
    git_host_url = git_status.build_url(file_path, line_number)
    if git_host_url:
        # This link is stored in the task definition
        source_code = task_definition_pb2.SourceCode(link=git_host_url)

Configuration Discovery​

Beyond source linking, the Git integration provides a utility for managing Flyte configuration. The config_from_root function allows developers to store their Flyte configuration within the repository and load it reliably regardless of their current working directory.

By default, it looks for a file at .flyte/config.yaml relative to the Git root:

def config_from_root(path: pathlib.Path | str = ".flyte/config.yaml") -> flyte.config.Config | None:
    # Uses git rev-parse --show-toplevel to find the root
    # Returns a Config object if the file exists

This is commonly used in local execution scripts to ensure the correct environment settings are applied:

if __name__ == "__main__":
    import flyte.git
    
    # Automatically find and use the config file at the git root
    flyte.init_from_config(flyte.git.config_from_root())

    run = flyte.run(main, x=10)

Extensibility​

The system is designed to be extensible via the GIT_URL_BUILDER_REGISTRY. While GitHub and GitLab are supported out-of-the-box, additional providers can be added by implementing the GitUrlBuilder protocol and registering the new host in src/flyte/git/_config.py. If a host is not recognized, GitStatus.build_url will log a warning and return an empty string, ensuring that deployment can continue even without source linking.