https://prefect.io logo
Title
r

Ralph Willgoss

04/17/2020, 9:34 AM
Hi All, really interested in Prefect and so far enjoyed playing with it. My questions is around tasks calling other tasks. Can you pass the flow context into other tasks, so that you can set dependencies or perform map functions? Or is it by design that all task orchestration must happen at the top level within the flow context? TaskA does 100 computations, where each computation needs info from TaskB which itself does 100 computations. I'm wanting to take advantage of Prefect distributed nature of tasks out of the box. So i can spin a cluster up and get at least 100 computations all happening on their own nodes.
j

Jeremiah

04/17/2020, 12:41 PM
Hi @Ralph Willgoss - at this time Prefect expects all orchestration to happen at the
Flow
level and all work at the
Task
level, so either we’d suggest an iterated map:
inputs = list(range(100))
b_result = task_b.map(inputs)
a_result = task_a.map(b_result)
or, if you don’t need prefect to govern the subtasks directly, you could farm all 100 out to something like your Dask cluster directly from inside a single task, then wait for the result, and let Prefect take orchestration over.
r

Ralph Willgoss

04/17/2020, 2:07 PM
yeah exactly what i thought. thanks very much as this is how I will treat it
@Jeremiah much appreciated, thanks!
j

Jeremiah

04/17/2020, 2:17 PM
Great!