I use a builder pattern to generate multiple flows...
# ask-community
a
I use a builder pattern to generate multiple flows with different structures based on some configs. With Prefect 1.0 I was able to deploy and run the flows locally but I'm having issues with the LocalStorage with Prefect2.0. Details in thread:
Copy code
from prefect import flow, task
from dataclasses import dataclass

from prefect.deployments import Deployment


@dataclass
class FlowBuildSpecs:
    name: str
    do_double: bool
    do_square: bool


@task
def double(x):
    return 2 * x


@task
def square(x):
    return x * x


def get_flow(specs):
    @flow(name=specs.name)
    def my_flow(x=5):
        val = x
        if specs.do_double:
            val = double(val)
        if specs.do_square:
            val = square(val)
        return val

    return my_flow


configs = [
    FlowBuildSpecs(name="a", do_double=True, do_square=True),
    FlowBuildSpecs(name="b", do_double=True, do_square=False),
]
for configs in configs[:1]:
    flow = get_flow(configs)

    deployment = Deployment.build_from_flow(
        flow=flow,
        name="local",
        tags=["test"],
        work_queue_name="test",
    )
    _id = deployment.apply()
    print(f"Deployment ID: {_id}")
I get this error
Copy code
prefect.exceptions.MissingFlowError: Flow function with name 'my_flow' not found in 'minimal_flow.py'.
Any ideas on how I can work with this pattern with Prefect 2.0?
k
Hi @alex, It looks like you are trying to create multiple deployments of the my_flow() and I think you don't actually need get_flow() with the configs. You can have multiple deployments for one flow and since you already have conditional logic for your flow with parameters, you can create two deployments for the same flow with just different parameters.
a
Hi Kalise, thanks for your reply. This was just a simplified example, in practice I have jobs running for multiple clients, each job has some configurations I use to disable certain tasks in the flow structure. The configurations aren't changed and they are plenty of them so I use this approach to avoid polluting the parameters and only keep ones we want to modify. My second concern is that I would like to be able to monitor jobs for each client, and based on the UI right now, it will be easier to get a high level overview of the historical progress when the jobs are separated as different flows vs different deployments
s
Hi @alex, Im in the same boat for this one (trying to build flows dynamically at runtime). Did you ever manage to solve this?
a
@Samuel Hinton unfortunately not, we had to to create a single root level flow and use the parameters to configure the steps for each deployment. There is an open issue you can follow for adding Pickle storage which is a requirement for being able to dynamically build flows.
s
Thanks for letting me know. I’ve added my use case for this to the issue at https://github.com/PrefectHQ/prefect/issues/8121.