hi there prefect 2 roadmap question is there some plan for a Prefect Community #ask-community

hi there! prefect 2 roadmap question: is there som...

Jai P

08/26/2022, 2:32 PM

hi there! prefect 2 roadmap question: is there some plan for a

DaskTaskRunner

, but for subflows? E.g. if i have a flow that kicks off a bunch of subflows, i want those to be distributed out and not running on the same instance as the parent flow (or...is that already possible and i'm missing something)? Thanks in advance!

✅ 1

Ryan Peden

08/26/2022, 6:19 PM

Hi Jai, I believe you can already this on a per-subflow basis. So, for example:

Copy code

from prefect import flow

@flow(task_runner=DaskTaskRunner(address="<http://my-dask-cluster-one>")
def subflow_one():
   print("hello from subflow one!")

@flow(task_runner=DaskTaskRunner(address="<http://my-dask-cluster-two>")
def subflow_two():
 print("hello from subflow two!")

@flow
def main_flow():
  subflow_one()
  subflow_two()

There's a bit more info in the composing flows section of the docs - if you scroll down a bit to the 'Subflows or tasks?' callout you well see a point about using different task runners for each subflow.

Jai P

08/26/2022, 6:38 PM

ah so the thing is, i'd like the subflow itself to be executed elsewhere...is that already happening?

Jai P

08/26/2022, 6:40 PM

for example:

Copy code

@flow
def my_subflow(arg):
    ....

@flow
def my_parent_flow(args):
    subflows = [my_subflow(arg) for arg in args]
    asyncio.gather(*subflows)

are triggered as co-routines on the same instance, but what i actually want is something like:

Copy code

@flow
def my_subflow(arg):
    ....

@flow
def my_parent_flow(args):
    subflows = [my_subflow.submit(arg) for arg in args]
    asyncio.gather(*subflows)

where the

.submit

call on the subflow actually triggers an api call to orion, a task is put on a work queue, and another instance picks it up

Jai P

08/26/2022, 6:41 PM

so perhaps

my_parent_flow

is triggered on

instance_1

, but

my_subflow(arg_1)

runs on

instance_2

my_subflow(arg_2)

runs on

instance_4

, etc.

Jai P

08/26/2022, 6:42 PM

perhaps that doesn't actually need to happen though...i never thought about whether simply having the tasks executing on a cluster is sufficient. I guess the only thing i'd want to guarantee is that i can trigger many subflow runs at the same time and parallelize the work

Ryan Peden

08/26/2022, 6:55 PM

That should work as long as your subflows are in async functions. I'll try to add an example to the docs - they mention running async subflows in parallel but don't have any code that demonstrates it

👍 1

3 Views

Open in Slack

Previous Next