Log Streaming and Visualization

To access real-time logs from remote Flyte executions, you can use the interactive log viewer for live debugging or stream raw log lines for programmatic processing.

View Interactive Logs

The most common way to view logs is through the show_logs method on an Action object. This creates a live-updating console view that scrolls as new logs arrive.

from flyte.remote._action import Action

# Get an action handle
action = Action.get(run_name="my-run-id", name="my-task-action")

# Display interactive logs (synchronous call)
action.show_logs(
    max_lines=50,
    show_ts=True,
    filter_system=True
)

The show_logs method uses the AsyncLogViewer class to manage a rich.live.Live display. If you are in a Jupyter notebook, it will attempt to use ipywidgets for a better experience; otherwise, it falls back to standard console output.

Stream Logs Programmatically

If you need to process log lines as strings (e.g., for searching or custom formatting), use the get_logs method. This returns an asynchronous generator.

import asyncio
from flyte.remote._action import Action

async def process_logs():
    action = await Action.get.aio(run_name="my-run-id", name="my-task-action")
    
    # Stream logs as they arrive
    async for line in action.get_logs(filter_system=True):
        if "ERROR" in line:
            print(f"Found error: {line}")

asyncio.run(process_logs())

Access Raw Log Protobufs

For low-level access to the log metadata (such as the originator or raw timestamps), use the Logs.tail method. This yields LogLine protobuf objects directly from the Flyte dataproxy service.

from flyte.remote._logs import Logs
from flyteidl2.logs.dataplane import payload_pb2

async def tail_raw_logs(action_id):
    # tail() yields payload_pb2.LogLine objects
    async for log_line in Logs.tail.aio(action_id=action_id, attempt=1):
        origin = payload_pb2.LogLineOriginator.Name(log_line.originator)
        print(f"[{origin}] {log_line.message}")

Customizing Log Output

The Logs class and Action methods provide several parameters to control how logs are filtered and displayed:

• filter_system: When True, it filters out logs where the originator is SYSTEM. It also hides internal Flyte logs (containing [flyte]) unless they contain flyte.errors.
• show_ts: When True, prefixes each log line with an ISO-8601 timestamp extracted from the LogLine protobuf.
• max_lines: (Viewer only) Limits the number of lines kept in the scrolling buffer.
• raw: (Viewer only) If set to True in show_logs, it bypasses the interactive Live display and prints lines directly to the console as they arrive.

Troubleshooting

Logs Not Yet Available

Logs may not be immediately available after an action starts. The Logs.tail method implements a retry mechanism that attempts to connect to the log stream 5 times with a 2-second sleep between attempts. If the stream still cannot be found, it raises a LogsNotYetAvailableError.

Jupyter Notebook Display

The interactive viewer requires ipywidgets to function correctly in Jupyter environments. If ipywidgets is missing, the SDK will log a warning and default to raw console output:

# Found in src/flyte/remote/_logs.py
if ipython_check():
    if not ipywidgets_check():
        logger.warning("IPython widgets is not available, defaulting to console output.")
        raw = True

Connection Errors

The log streamer handles ConnectError from the underlying RPC. If a NOT_FOUND (404) error occurs, it is treated as a "not yet ready" state and triggers the retry logic. Other connection errors are re-raised immediately.