How do you people supplying parameters for a flow for different environments e.g. local, staging and prod? With CLI-based deployment, I have to modify deployment files locally or modify in web ui after applying. I think it's weird to drop the

Deployment

class. It can provide the deployment configuration above the

DO NOT EDIT

line in a generated deployment file. https://orion-docs.prefect.io/concepts/deployments/

Would you want to have dev and prod deployments within the same workspace or Orion instance? My recommendation would be the same as AWS best practice to leverage separate accounts, workspaces and environments but I totally understand why different patterns may suit different users and we'll provide more guidance on that in the future

Thx Anna, but I meant about CI/CD automation. Since the deployment process is: 1. prefect deployment build 2. manually edit input parameters in a generated deployment file 3. prefect deployment apply I want to get rid of the manual process (2). I wish to have deployment configurations for each environment stored in git.

/
|- flows
   |- my-awesome-flow
      |- my-awesome-flow.py
      |- deployment-staging.yaml
      |- deployment-production.yaml
|- out
   |- my-awesome-flow-staging.yaml
   |- my-awesome-flow-staging-manifest.json

The

deployment-staging.yml

has the configurations above the

DO NOT EDIT

line.

name: my-awesome-flow-staging
description: null
version: very_long_hash
parameters: {"param_a": "A"}

The

my-awesome-flow-staging.yaml

is generated with the above configuration.

name: my-awesome-flow-staging
description: null
version: very_long_hash
parameters: {"param_a": "A"}
...
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: my-awesome-flow
...

This is the simplest setup if you would want dev and prod deployments within the same env:

prefect deployment build flows/hello.py:hello --name dev --tag dev -sb s3/dev --output deployments/hello-dev.yaml
prefect deployment build flows/hello.py:hello --name prod --tag prod -sb s3/prod --output deployments/hello-prod.yaml

Let me try the s3 block a bit more. I'm worried about the hashes below the

DO NOT EDIT

line (based on what I see in local deploy). It would be nice if a generated deployment file is created once and modified many times without worrying those hashes. Then, I can git the file.

There is actually no need to version control the YAML manifest unless you want to. It's best to treat those as deployment artifacts. The CLI command and resulting final deployment representation in the API is what matters and the YAML definition is the final representation that you can use for sanity check that your API-managed deployment entity looks as you wanted to. Again, I encourage you to check #CKNSX5WG3 where I explained and asked to wait before building a deployment strategy as we are currently working on improving that experience and making it less ambiguous

Well… I personally would have liked to be able to define deployments in a YAML file (and version control them). This way, when you open up someone else's flow repo, you also see how they also deploy their flows. But then again, I come from a k8s background so I use YAML all the time with

kubectl apply

. Something similar with Prefect would have been nice:

prefect deployment build

to help you generate the deployment and

prefect deployment apply manifest.yaml

to create the deployment (move file to storage location, create the representation in the API etc).

@Matthias that's definitely possible. The only challenge with this is e.g. if you generate a Docker image from a CI/CD pipeline and want to put the new image tagged with your Git commit sha, then this pattern doesn't work. But if you keep a static image tag, the approach you mentioned works 100%, same as with Kubernetes. As always tradeoffs with all approaches, but what you described is definitely supported and possible

Recently moving from 2.0b7 to 2.0.2, I've faced the major change of the deployment workflow. I know you’re working on it, but are some thoughts and problems with the current release. As Pipa, I liked to be able to defined the deployments from python in 2.0b7, because I've many deployments (~5 flows, and 4 different values for the parametre, and I’m only at the beginning of my pipeline). I had a basic loop to generate the deployment for each case in python. The scheduled time between the flows was also calculated in that python code (like this flow with this parameter run 5 min after this flow with that parameter). Obviously with the new system (only using CLI is documented) I could instead write a bash script to achieve the same goal, or use python with os.system. That's not convenient and not as nice as before. I like to write the tasks, the flows and the deployments in the same language, in the same well design perfect spirit. I'm pretty sure that there is a pure python solution under the hood, my first question is whether such option will be documented/official/easy-to-use or not. The second question is about an incompatibility between the new workflow and debugging. Since I can't run my flows locally on my laptop (datalake is not accessible from home where I remote work), I'm editing locally the python flow code but deploy for every small change on a test server. It means tens/hundreds of deployments per day for the same script (actually I'm not a pro developer and my server has very little memory w/r to the data processing → a lot of fine tuning of my code). With the current deployment work, I have to build & apply the deployment for every change in the code and since the parameter and scheduler are not persisted in this file and they can not be set with the CLI, I'd have to edit the .yaml file everything. This is the step 2 described by Pipa. Currently, I’ve set default python arguments for all my flow for this reason, but this is not practical in the long term. I’m wondering if there is another way to work for debugging remotely ? At the very minimum, I’d like to the set the parameters with the CLI, or to allow build+upload+apply to be separate steps. And ideally still be able to define the deployment using python.

For this remote flow run execution, you could either SSH to this server and run your flows directly, or you could have a remote agent the best thing about remote storage in the current deployment UX is that it also packages your custom modules so even if your flow code lives on your machine and you then want to run it on a different VM, it will just work because Prefect will grab the full project from S3 (not only your flow code) for schedule and parameters there will most likely be CLI flags so no need to edit it via YAML - all WIP

Yes, I'm working with a remote storage and an agent on the server (+orion), and the packaging is a big recent advance (I had to duplicate code in the past, now I'm using modules). This is very convenient to 'synchronise' the local code to the server (via the remote storage). What seems to be missing is 1) to do the same in python as with the cli and/or 2) to allow to upload code without building the development file (unless you consider that editing the file is bad practice?). As deployment cli has 'build' and 'apply', which are separte (for a reason I don't understand), it would be useful to also have 'upload' or 'update'.

Very soon you'll be able to configure everything from CLI and the YAML file is basically a reference and confirmation showing how the deployment looks like It will be clearer Soon

@Anna Geller how to set the flow parameters by command line...I think this is not possible yet, right? How to set the parameters within the ci/cd deployment... I thought about running

prefect deployment build

while executing the ci/cd pipeline and appyling the deployments to Orion. The deployment yaml files are threated as artifacts like you said and not included in version control.

This is still a work in progress - not possible via CLI yet, but we are working on it

Thanks