https://prefect.io logo
Powered by
Title
j

Jai P

08/26/2022, 2:32 PM
hi there! prefect 2 roadmap question: is there some plan for a 
DaskTaskRunner
, but for subflows? E.g. if i have a flow that kicks off a bunch of subflows, i want those to be distributed out and not running on the same instance as the parent flow (or...is that already possible and i'm missing something)? Thanks in advance!
✅ 1
r

Ryan Peden

08/26/2022, 6:19 PM
Hi Jai, I believe you can already this on a per-subflow basis. So, for example: 
from prefect import flow

@flow(task_runner=DaskTaskRunner(address="<http://my-dask-cluster-one>")
def subflow_one():
   print("hello from subflow one!")

@flow(task_runner=DaskTaskRunner(address="<http://my-dask-cluster-two>")
def subflow_two():
 print("hello from subflow two!")

@flow
def main_flow():
  subflow_one()
  subflow_two()
There's a bit more info in the composing flows section of the docs - if you scroll down a bit to the 'Subflows or tasks?' callout you well see a point about using different task runners for each subflow.
j

Jai P

08/26/2022, 6:38 PM
ah so the thing is, i'd like the subflow itself to be executed elsewhere...is that already happening?
for example: 
@flow
def my_subflow(arg):
    ....

@flow
def my_parent_flow(args):
    subflows = [my_subflow(arg) for arg in args]
    asyncio.gather(*subflows)
are triggered as co-routines on the same instance, but what i actually want is something like: 
@flow
def my_subflow(arg):
    ....

@flow
def my_parent_flow(args):
    subflows = [my_subflow.submit(arg) for arg in args]
    asyncio.gather(*subflows)
where the 
.submit
call on the subflow actually triggers an api call to orion, a task is put on a work queue, and another instance picks it up
so perhaps 
my_parent_flow
is triggered on 
instance_1
, but 
my_subflow(arg_1)
runs on 
instance_2
, 
my_subflow(arg_2)
runs on 
instance_4
, etc.
perhaps that doesn't actually need to happen though...i never thought about whether simply having the tasks executing on a cluster is sufficient. I guess the only thing i'd want to guarantee is that i can trigger many subflow runs at the same time and parallelize the work
r

Ryan Peden

08/26/2022, 6:55 PM
That should work as long as your subflows are in async functions. I'll try to add an example to the docs - they mention running async subflows in parallel but don't have any code that demonstrates it
👍 1
3 Views
#prefect-communityJoin Slack
Powered by