https://prefect.io
Join Slack
Hello Community, We have around 50 deployments in our prefect project, each deployment has got a bi...
# ask-community
h
Hello Community, We have around 50 deployments in our prefect project, each deployment has got a big/long number of parameters. We used to implement all deployment using the 
Deployment
stack, each deployment in its own file, each file has a 
deployment.apply()
which would deploy it when run. We are currently migrating to using 
prefect.yaml
(our understanding is that 
Deployment
is getting deprecated). My question is: is it normal to put all deployments in same file? this would lead to a huge yaml file that could easily become unmaintainable, if not, what do you guys use as patterns to circumvent this?
y
@Hamza Mogni We solved this by putting a comment part in the top of each flow:
CleanShot 2024-12-26 at 15.49.55@2x.png
Then we have a create_prefect_yaml_from_comments.py script that goes over all the files, and creates the prefect.yaml
Than we just run 
prefect --no-prompt deploy --all
as usual, as we've just programatically created the yaml file.
Regarding create_prefect_yaml_from_comments.py, it's very easy. You can even use chatgpt to quickly create it.
👍 1
h
interesting, thanks. By curiosity, how do you manage flows that have multiple deployments linked to it?
one of our thoughts was to create multiple yaml files for deployments and use a python script to compile them to prefect.yaml . your approach seems to confirm this one
y
We are not using the doing things like one flow -> many deployments.
They are always one to one
Maybe it's for advanced use case, but we found it very confusing and not helpful.
n
hi @Hamza Mogni - there's a --prefect-file flag for 
prefect deploy
so you dont have to keep all your definitions in the same yaml file
would be happy to chat more about this question
how do you manage flows that have multiple deployments linked to it?
im not sure I understand the question, you can always create many deployments from one flow - what do you see as the organizational challenge as it relates to that?
h
Hello @Nate yes we are creating multiple flows for same deployments, i was only curious how they manage it using that approach (docstrings). yes I tried using 
--prefect-file
but my understanding is that it only takes one file path as an argument, meaning there would be a need for a preprocessing step to merge definitions from separate yaml files into a single one and then feed the output to --prefect-file.
n
aha yes you're correct if deployments in all files would depend on the same definitions that would indeed be required alternatively you can use the python interface if you want to employ a modular approach to apply common deployment config and organize it however you can organize python code
👍 1
it feels like there could also be a reasonable enhancement request related to "global `definitions`" or something like that that could be referenced in any deployment related to a given server / workspace (regardless of file)
h
ah yes, a global definitions approach would be very nice, possibly in a separate file like 
global-definitions.yaml
which would be automatically picked up by prefect before applying deployments from different yaml files
n
yeah that sounds very doable. are you open to creating an enhancement issue for this?
h
yup, would like to try and work on it also but don't know where to start, might take me some considerable time to dig it
n
no worries! just articulating the problem in an issue would be helpful but i can do so later if not
h
yes will definitely create the issue as a first step 👌
🙏 1
Hello @Nate I have created the issue here
19 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by