Samuel Hinton
09/16/2024, 2:23 AMdeployments = []
for our_flow in list_of_our_flows:
flow_from_source = await our_flow.from_source(repo, entrypoint=entrypoint) # Takes 2s to run
deployments.append(await flow_from_source.to_deployment(...)) # Takes 4s to run
await prefect.deploy(*deployments, ...)
With about 100 flows/deployment combinations, that's 10 minutes of registering flows, all belonging to the same repo, and given we have a dev env before prod (ie we deploy twice), our pipeline to update a flow has broken 20minutes.
Does anyone know either why making a deployment takes multiple seconds, or if theres a way of doing this differently or somehow moving the <http://flow_from_source.to|flow_from_source.to>_deployment
into a batch?
I'm also not sure on best practises here, like should I do what I found in one tutorial (to_deployment and then prefect.deploy, or just flow.deploy, or something else). Our use case is we have one repo, which contains 100 or so flows, each that have a RRule pattern on them, and we want to make one deployment per flow+schedule pair. It seems like cloning things 100 times isnt ideal, but it seems to be what the documentation suggests?Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by