https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
s

Sean Talia

01/12/2023, 2:22 PM
hello! Has anyone ever had success with Prefect v1 in creating a flow that dynamically creates a number of 
StartFlowRun
tasks at runtime? My scenario is that I want a flow (call it 
orchestratorFlow
) to orchestrate the execution of several instances of another flow that I have (call that one 
ecsFlow
). The issue at hand is, I don't know how many instances of my 
ecsFlow
I need to kick off until 
orchestratorFlow
runtime; it could be 1, 8, or 50+ (each flow run of 
ecsFlow
would be run with a different set of parameters), and would be determined based on a parameter passed to 
orchestratorFlow
. I've tried to follow the Prefect v1 map paradigm by doing 
StartFlowRun.map([<config_parameters>])
, where 
[<config_parameters>]
is a dynamically generated list of flow run kwargs returned by an upstream task, but that's not doing the trick. Thank you for any help!
👀 1
m

Matthew Scanlon

01/12/2023, 3:31 PM
have you tried using 
prefect.tasks.prefect.create_flow_run
?
j

James Sopkin

01/12/2023, 3:37 PM
yes, I have done this before. What you can do is have your 
orchestrator flow
iterate through the parameters, kicking off a new child flow for each set of params using 
create_flow_run
. I can type up some pseudo code if that would help
I think it would look something like this
Copy code
from prefect import Flow, task, Client, unmapped


@task
def generate_params():
    # do something here
    return [{"num": 1, "letter":"a"}, {"num":2, "letter":"b"}, {"num":3, "letter":"c"}]


with Flow("orchestrator") as flow:
    params = generate_params()
    Client.create_flow_run.map(flow_id=unmapped("flow_id_here"), params=params)
s

Sean Talia

01/12/2023, 3:51 PM
oh okay interesting; it's actually unclear to me when you'd want to use 
create_flow_run
v. 
StartFlowRun
, the documentation isn't particularly clarifying...also does the different casing on these tasks signify anything?
j

James Sopkin

01/12/2023, 4:06 PM
I believe create_flow_run is an update on StartFlowRun
🙌 1
What do you mean by the different casing?
s

Sean Talia

01/12/2023, 4:07 PM
just the fact that 
StartFlowRun
is camel case and 
create_flow_run
is snake case
j

James Sopkin

01/12/2023, 4:08 PM
ahh
I'm not sure on the significance of this
👍 1
s

Sean Talia

01/12/2023, 4:08 PM
is snake case the new standard and 
StartFlowRun
is just kept to preserve backwards compatibility?
j

James Sopkin

01/12/2023, 4:08 PM
I believe that is the case
3 Views
Powered by