Standard Python Collections

Flyte handles standard Python containers—lists, dictionaries, tuples, and TypedDicts—by mapping them to specific Flyte IDL types through specialized transformers. These transformers ensure that complex Python structures are correctly serialized into Flyte Literals and reconstructed back into Python objects during task execution.

Lists and Univariate Collections

The ListTransformer (found in src/flyte/types/_type_engine.py) handles typing.List[T]. Flyte requires that lists are univariate, meaning all elements must share the same generic type T.

Mapping to Flyte IDL

A Python list is transformed into a LiteralCollection. The ListTransformer recursively calls the TypeEngine to convert each element based on the sub-type T.

# Example of a typed list in a Flyte task
@task
def process_scores(scores: typing.List[float]) -> float:
    return sum(scores) / len(scores)

Async Transformation

To optimize performance for large collections, ListTransformer processes element transformations in parallel using _run_coros_in_chunks. The batch size is controlled by _TYPE_ENGINE_COROS_BATCH_SIZE, allowing Flyte to handle large lists without blocking the event loop.

Dictionaries and Maps

The DictTransformer (in src/flyte/types/_type_engine.py) handles Python dictionaries. It supports two distinct serialization paths depending on the key type and type hints.

Typed Maps (String Keys)

If a dictionary is hinted as typing.Dict[str, T], Flyte maps it to a LiteralMap. This is the most efficient representation and is natively supported by the Flyte Console.

@task
def get_metadata() -> typing.Dict[str, int]:
    return {"version": 1, "count": 42}

Binary Structs and Serialization Fallbacks

If the dictionary uses non-string keys (e.g., Dict[int, str]) or is untyped, Flyte treats it as a STRUCT. The DictTransformer attempts to serialize these using MessagePack via the MessagePackEncoder.

If MessagePack encoding fails and the type is annotated with allow_pickle, Flyte falls back to Pickle serialization, storing the result via FlytePickle.

# Dictionary with non-string keys using MessagePack
@task
def map_ids(mapping: typing.Dict[int, str]) -> str:
    return mapping.get(1, "unknown")

Structured Collections via Pydantic Wrapping

For more complex structures like tuple, NamedTuple, and TypedDict, Flyte uses a "wrapping" strategy implemented in src/flyte/types/_tuple_dict.py. These transformers inherit from PydanticWrappingTransformer, which dynamically generates a Pydantic BaseModel to represent the collection's structure.

Tuples and NamedTuples

TupleTransformer and NamedTupleTransformer handle fixed-size ordered collections.

• Tuples: Elements are mapped to fields named item_0, item_1, etc.
• NamedTuples: Field names are preserved from the NamedTuple definition.

# Tuple transformation example
@pytest.mark.asyncio
async def test_tuple_of_primitives(self):
    pt = tuple[int, str, float, bool]
    value = (42, "hello", 3.14, True)
    
    # Internally, TupleTransformer creates a Pydantic model:
    # class TupleWrapper_tuple(BaseModel):
    #     item_0: int
    #     item_1: str
    #     ...

TypedDicts

The TypedDictTransformer provides robust validation for TypedDict types. It specifically handles:

• Required/NotRequired: It respects typing.Required and typing.NotRequired hints to determine if keys must be present.
• Self-Referential Types: It maintains an instance-level _model_cache to handle recursive definitions (e.g., a tree node containing a list of tree nodes).

class PersonInfo(TypedDict):
    name: str
    age: int
    email: str

@task
def greet(person: PersonInfo) -> str:
    return f"Hello {person['name']}"

Schema Reconstruction

When Flyte receives a Literal of type STRUCT, these transformers can "guess" the original Python type by inspecting the JSON schema metadata. For example, TupleTransformer looks for a TupleWrapper_ prefix in the schema title to reconstruct the original tuple[...] type hint.

Performance Considerations

Both ListTransformer and DictTransformer utilize asynchronous processing for their elements. The _run_coros_in_chunks utility ensures that even deeply nested or very large collections do not exhaust system resources by limiting the number of concurrent transformation coroutines.

Transformer	Flyte IDL Type	Serialization Format
ListTransformer	LiteralCollection	Recursive IDL
DictTransformer (str keys)	LiteralMap	Recursive IDL
DictTransformer (other)	STRUCT	MessagePack / Pickle
TupleTransformer	STRUCT	Pydantic / MessagePack
TypedDictTransformer	STRUCT	Pydantic / MessagePack