Ralph Willgoss
04/17/2020, 9:34 AMJeremiah
04/17/2020, 12:41 PMFlow
level and all work at the Task
level, so either we’d suggest an iterated map:
inputs = list(range(100))
b_result = task_b.map(inputs)
a_result = task_a.map(b_result)
or, if you don’t need prefect to govern the subtasks directly, you could farm all 100 out to something like your Dask cluster directly from inside a single task, then wait for the result, and let Prefect take orchestration over.Ralph Willgoss
04/17/2020, 2:07 PMJeremiah
04/17/2020, 2:17 PM