i have a flow that i want to dynamically name, sch...
# prefect-community
j
i have a flow that i want to dynamically name, schedule, and pass parameters to, how do i do that?
1️⃣ 1
m
Hey Jon when you're saying dynamically name are your referring to the Flow Name or the Flow Run Name, The Flow name is determined at registration with 1.0 so if that's the case you'd technically be registering flows multiple times, the flow run name can be set dynamically https://discourse.prefect.io/t/how-can-i-set-a-custom-flow-run-name-instead-of-using-the-randomly-generated-adjective-noun-combination/131. This might also be a good article to check out in terms of dynamic schedules/parameters https://discourse.prefect.io/t/one-flow-with-multiple-parameter-sets-and-schedules/693 Overall what are you wanting to achieve with the dynamic values?
upvote 1
j
maybe i don't have the prefect flow model in my head right, but basically i want the ability to run the same flow with different sets of parameters on different schedules, and to be able to turn any one of them off via the UI at any given time
so my thought is to name the flow like
flow_customer_a
,
flow_customer_b
, etc ... when it's all the same underlying flow, just different configs passed in doing slightly different stuff behind the scenes
someone from the prefect team posted about deployment docs that address this. but seems to have been deleted. mind sharing? also: when a flow is wrapped in a register function, how to call it independently and test it?
m
Deployments are a 2.0 concept https://docs.prefect.io/concepts/deployments/, this is once of the use cases we wanted to solve for in comparison to 1.0. The Schedule in particular is hard to address dynamically though the Discourse above is a good baseline to start from. Basically because everything is predefined at build/registration time for 1.0 it really makes these kinds of dynamic use case difficult to achieve since you can't really parameterize the schedules. Other options might be to leverage an reference to determine the parameters at runtime within a task, something like the kv store or some other data table that a task can reference. The schedules though are a bit more difficult generally speaking you can have multiple schedules on the same flow though but updating a schedule for a flow within the flow while it's running isn't going to really have an impact on the current run. You might also look into more of an orchestrator pattern here with a parent flow that calls different flows based on some predefined condition, though again in 1.0 this is still likely going to require some external reference or you'd need to determine the exact parameters you need within that task rather than when the flow. TLDR: Dynamic Schedules/parameters are difficult to accommodate in 1.0 due to the nature of a DAG, we address this with deployments in 2.0 to make it more achievable.
j
Basically because everything is predefined at build/registration time for 1.0 it really makes these kinds of dynamic use case difficult to achieve since you can't really parameterize the schedules.
i don't follow this. what is wrong about this approach in v1?:
Copy code
def register_flow(flow_name, schedule, param):
    with Flow(name=flow_name, schedule=schedule) as flow:
         task_result = some_task(param)
if we give it an entirely different name for each call of regitser_flow() do we get around what you're describing?
thx for the detailed response btw 🙂
m
There's nothing inherently wrong with that approach it will just register multiple flows which will behave independently where as deployments in 2.0 allow you to have a single flow that is tied to multiple deployments each with their own parameters, schedules, dependencies, etc. Basically the approach is just cumbersome in 1.0 due to the inherent nature of having to structure everything around a DAG whereas you have significantly more wiggle room in 2.0 if that makes sense I'd definitely include at least some recognizable identifier on the flow registration. No Problem 😄
j
register multiple flows which will behave independently where as deployments in 2.0 allow you to have a single flow that is tied to multiple deployments
trying to wrap my mind around this. can you give me an example use case for the 1.0 approach vs 2.0 approach? in our case, we want to deploy and run entirely separate flows, depending on the parameters passed to the same flow "template".
I'd definitely include at least some recognizable identifier on the flow registration.
what do you mean? how is this different than giving it a unique name?
m
For sure this Article I sent before pretty much outlines the 1.0 approach One Flow with Multiple Parameter Sets and Schedules the big drawback here being that each registered flow will treated independently by our system so depending on your use case you could end up with dozens of versions of the same flow. as for 2.0 you could do something like this
Copy code
@flow
def Parent_flow(deployment, params, start_time):
    run_deployment(
            name=deployment, 
            parameters=params,
            scheduled_time=start_time)
or this:
Copy code
@flow
def Parent_flow(params, start_time):
    if some_condition:
        run_deployment(
            name="flow-name/deploymen-name", 
            parameters=params,
            scheduled_time=start_time)
    else some_other_condition:
        run_deployment(
            name="flow-name/deploymen-nameb", 
            parameters=params,
            scheduled_time=start_time)
The Deployment, params, start_time that you're referencing for this flow could be for a single deployed flow or multiple depending on the specifics of what you're wanting to achieve, deployments is a pretty vast concept in 2.0 so it's difficult to cover all of the specifics since there's a lot of variance depending on what you're trying to achieve. It's worth noting that you may not even need multiple deployments with this kind of pattern since you can specify the params and schedule dynamically with the run_deployment command. The only time you might need different deployments here would likely be if different scenarios required different external dependencies 😄
This is also just one way of achieving this, you may not even need to use the run deployment command if you just need some conditional logic around what runs within a flow depending on the params
j
yeah im on prefect 1 and just want to have separate flows for each config i pass in, like in the code snippet i showed above