https://prefect.io logo
Title
p

Pipat (Benz) Methavanitpong

08/04/2022, 3:58 AM
How do you people supplying parameters for a flow for different environments e.g. local, staging and prod? With CLI-based deployment, I have to modify deployment files locally or modify in web ui after applying. I think it's weird to drop the
Deployment
class. It can provide the deployment configuration above the
DO NOT EDIT
line in a generated deployment file. https://orion-docs.prefect.io/concepts/deployments/
4
a

Anna Geller

08/04/2022, 8:06 AM
Would you want to have dev and prod deployments within the same workspace or Orion instance? My recommendation would be the same as AWS best practice to leverage separate accounts, workspaces and environments but I totally understand why different patterns may suit different users and we'll provide more guidance on that in the future
p

Pipat (Benz) Methavanitpong

08/04/2022, 8:21 AM
Thx Anna, but I meant about CI/CD automation. Since the deployment process is: 1. prefect deployment build 2. manually edit input parameters in a generated deployment file 3. prefect deployment apply I want to get rid of the manual process (2). I wish to have deployment configurations for each environment stored in git.
/
|- flows
   |- my-awesome-flow
      |- my-awesome-flow.py
      |- deployment-staging.yaml
      |- deployment-production.yaml
|- out
   |- my-awesome-flow-staging.yaml
   |- my-awesome-flow-staging-manifest.json
The
deployment-staging.yml
has the configurations above the
DO NOT EDIT
line.
name: my-awesome-flow-staging
description: null
version: very_long_hash
parameters: {"param_a": "A"}
The
my-awesome-flow-staging.yaml
is generated with the above configuration.
name: my-awesome-flow-staging
description: null
version: very_long_hash
parameters: {"param_a": "A"}
...
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: my-awesome-flow
...
1
a

Anna Geller

08/04/2022, 8:42 AM
This is the simplest setup if you would want dev and prod deployments within the same env:
prefect deployment build flows/hello.py:hello --name dev --tag dev -sb s3/dev --output deployments/hello-dev.yaml
prefect deployment build flows/hello.py:hello --name prod --tag prod -sb s3/prod --output deployments/hello-prod.yaml
no manual process is needed, you just need separate deployments for different environments
a bit of a spoiler but the JSON file will go away soon, all the info will move into YAML to have only 1 manifest per flow deployment making things less confusing
🙌 1
🙌🏻 1
p

Pipat (Benz) Methavanitpong

08/04/2022, 9:01 AM
Let me try the s3 block a bit more. I'm worried about the hashes below the
DO NOT EDIT
line (based on what I see in local deploy). It would be nice if a generated deployment file is created once and modified many times without worrying those hashes. Then, I can git the file.
a

Anna Geller

08/04/2022, 9:38 AM
There is actually no need to version control the YAML manifest unless you want to. It's best to treat those as deployment artifacts. The CLI command and resulting final deployment representation in the API is what matters and the YAML definition is the final representation that you can use for sanity check that your API-managed deployment entity looks as you wanted to. Again, I encourage you to check #announcements where I explained and asked to wait before building a deployment strategy as we are currently working on improving that experience and making it less ambiguous
m

Matthias

08/04/2022, 6:00 PM
Well… I personally would have liked to be able to define deployments in a YAML file (and version control them). This way, when you open up someone else's flow repo, you also see how they also deploy their flows. But then again, I come from a k8s background so I use YAML all the time with
kubectl apply
. Something similar with Prefect would have been nice:
prefect deployment build
to help you generate the deployment and
prefect deployment apply manifest.yaml
to create the deployment (move file to storage location, create the representation in the API etc).
a

Anna Geller

08/04/2022, 7:34 PM
@Matthias that's definitely possible. The only challenge with this is e.g. if you generate a Docker image from a CI/CD pipeline and want to put the new image tagged with your Git commit sha, then this pattern doesn't work. But if you keep a static image tag, the approach you mentioned works 100%, same as with Kubernetes. As always tradeoffs with all approaches, but what you described is definitely supported and possible
g

Ghislain Picard

08/07/2022, 4:26 PM
Recently moving from 2.0b7 to 2.0.2, I've faced the major change of the deployment workflow. I know you’re working on it, but are some thoughts and problems with the current release. As Pipa, I liked to be able to defined the deployments from python in 2.0b7, because I've many deployments (~5 flows, and 4 different values for the parametre, and I’m only at the beginning of my pipeline). I had a basic loop to generate the deployment for each case in python. The scheduled time between the flows was also calculated in that python code (like this flow with this parameter run 5 min after this flow with that parameter). Obviously with the new system (only using CLI is documented) I could instead write a bash script to achieve the same goal, or use python with os.system. That's not convenient and not as nice as before. I like to write the tasks, the flows and the deployments in the same language, in the same well design perfect spirit. I'm pretty sure that there is a pure python solution under the hood, my first question is whether such option will be documented/official/easy-to-use or not. The second question is about an incompatibility between the new workflow and debugging. Since I can't run my flows locally on my laptop (datalake is not accessible from home where I remote work), I'm editing locally the python flow code but deploy for every small change on a test server. It means tens/hundreds of deployments per day for the same script (actually I'm not a pro developer and my server has very little memory w/r to the data processing → a lot of fine tuning of my code). With the current deployment work, I have to build & apply the deployment for every change in the code and since the parameter and scheduler are not persisted in this file and they can not be set with the CLI, I'd have to edit the .yaml file everything. This is the step 2 described by Pipa. Currently, I’ve set default python arguments for all my flow for this reason, but this is not practical in the long term. I’m wondering if there is another way to work for debugging remotely ? At the very minimum, I’d like to the set the parameters with the CLI, or to allow build+upload+apply to be separate steps. And ideally still be able to define the deployment using python.
a

Anna Geller

08/07/2022, 6:06 PM
For this remote flow run execution, you could either SSH to this server and run your flows directly, or you could have a remote agent the best thing about remote storage in the current deployment UX is that it also packages your custom modules so even if your flow code lives on your machine and you then want to run it on a different VM, it will just work because Prefect will grab the full project from S3 (not only your flow code) for schedule and parameters there will most likely be CLI flags so no need to edit it via YAML - all WIP
g

Ghislain Picard

08/07/2022, 7:10 PM
Yes, I'm working with a remote storage and an agent on the server (+orion), and the packaging is a big recent advance (I had to duplicate code in the past, now I'm using modules). This is very convenient to 'synchronise' the local code to the server (via the remote storage). What seems to be missing is 1) to do the same in python as with the cli and/or 2) to allow to upload code without building the development file (unless you consider that editing the file is bad practice?). As deployment cli has 'build' and 'apply', which are separte (for a reason I don't understand), it would be useful to also have 'upload' or 'update'.
a

Anna Geller

08/07/2022, 7:17 PM
Very soon you'll be able to configure everything from CLI and the YAML file is basically a reference and confirmation showing how the deployment looks like It will be clearer Soon
🙏 2
t

Timo

08/30/2022, 7:59 AM
@Anna Geller how to set the flow parameters by command line...I think this is not possible yet, right? How to set the parameters within the ci/cd deployment... I thought about running
prefect deployment build
while executing the ci/cd pipeline and appyling the deployments to Orion. The deployment yaml files are threated as artifacts like you said and not included in version control.
a

Anna Geller

08/30/2022, 11:10 AM
This is still a work in progress - not possible via CLI yet, but we are working on it
For now, you can set your default parameter values directly on your flow function and when you need to later override it, you can do it from the UI
👍 1
t

Timo

08/30/2022, 11:19 AM
Thanks
🙌 1