👋🏻 Hi all. Looking for some general guidance - does anyone have any best practices when it comes to deploying many (100+) jobs? For a while I've been using a

prefect.yaml

deployment config for a dozen jobs which has worked well. Coming up, I'm looking to port over a huge pile of older, unmanged pipelines into Prefect. Right now each deployment config in my YAML is about 13 lines. Feels like there has to be a more efficient way to do this rather than configuring 1300 lines where each job has a different name, schedule, and ~2 parameters. Maybe a script to take a flattened input and re-create the prefect YAML every time I deploy?

hi @Eric Albanese - do you want to keep everything in yaml? if you're planning on scripting to gather / setup / define deployments, you way want to consider the

.deploy()

method on flows or instead the

deploy(*many_flows)

util

@Nate Pretty flexible on the format! Looking at the Docs now.. this is the page I'm looking for right?

yep! https://docs.prefect.io/3.0/deploy/infrastructure-concepts/deploy-via-python#deploy-multiple-flows

Gotcha.... so this would be a stand alone deployment file that handles all that. Then presumably I could set a loop to read through some config parameters and set them to the flows. One last Q. I have my Flows set up in separate files. I'm thinking I'd just Import them and then config? So maybe something like

from prefect import flow, deploy
from flows import flow1, flow2

if __name__ == "__main__":
    deploy(
        flow1.to_deployment("my-deployment-1"),
        flow2.to_deployment("my-deployment-2"),
)

yep, thats exactly what i was imagining based on your description

great, giving this a spin now. Appreciate the quick info Nate

Hmm getting stuck on this. Does this approach always require creating a new Docker container? I already have a container image that's ready to go. Just trying to get something working, I have this so far

var_config = {
            "network_mode": "host",
            "volumes": [
                "/var/run/docker.sock:/var/run/docker.sock:ro",
             "/home/ubuntu/.docker/config.json:/root/.docker/config.json:ro"
                ]
            }

    my_flow_1.deploy(
        name='test-my_flow',
        image="xxxx/prefect:v10284.0",
        push=False,
        work_pool_name="ec2-docker",  
        job_variables=var_config
    )
    print(f"Deployed to Prefect Cloud")

oh i think you want

build=False

ok cool

being a bit pedantic, but i wouldnt mess with the actual entrypoint to the container, since if you're running flows in most cases you want the prefect engine to handle that i assume your code is baked into the image? or pulling down at runtime

baked into the image

cool then yeah its just a matter of setting the right

set_working_directory

args, which in the python way you can do something like

deploy(*[f.from_source(..).to_deployment(...) for f, config in zip(flows, configs)])

where

from_source

will get translated to a

set_working_directory

step in the

pull

section if you pass

source=str(Path(...))

and

entrypoint='filename.py:decorated_fn'

like this