Kyle McChesney
04/15/2026, 4:40 PMMarvin
04/15/2026, 4:40 PMMarvin
04/15/2026, 4:42 PMfrom prefect import task, flow
@task
def read_lines(path: str):
with open(path) as f:
for line in f:
yield line.rstrip("\n")
@flow
def pipeline():
for line in read_lines("huge.txt"):
# process each line without loading all into memory
print(line)
- Parallel processing with mapping
from prefect import task, flow
@task
def process_item(x: int) -> int:
return x * 2
@flow
def pipeline():
items = range(1_000_000) # concrete iterable
futures = process_item.map(items) # concurrent execution
results = [f.result() for f in futures]
- Avoid returning a raw generator (it becomes a list anyway)
@task
def bad():
return (i * 2 for i in range(1000)) # will be auto-materialized to a list
Docs and references
- Write and run workflows (tasks, generators): https://docs.prefect.io/v3/how-to-guides/workflows/write-and-run#create-tasks-and-child-workflows
- Running work concurrently (map): https://docs.prefect.io/v3/how-to-guides/workflows/run-work-concurrently#using-map
- Result persistence and serialization: https://docs.prefect.io/v3/advanced/results
- Task engine generator internals (API reference):
- https://docs.prefect.io/api-ref/python/prefect-task_engine/#prefect.task_engine.run_generator_task_sync
- https://docs.prefect.io/api-ref/python/prefect-task_engine/#prefect.task_engine.run_generator_task_async
If you can share your specific use case (streaming vs. persisting, size of data, need for concurrency), I can suggest the best pattern and a concrete snippet.