Kiley Roberson
08/01/2023, 10:27 PMpull
defined in the yaml but also adds an additional step (For example runs a run_shell_script command)? And if so what would the syntax for that be?Marvin
08/01/2023, 10:28 PMprefect.yaml
that uses the default pull
defined in the yaml and also adds an additional step. The pull section is the most important section within the prefect.yaml
file as it contains instructions for preparing your flows for a deployment run. These instructions will be executed each time a deployment created within this folder is run via a worker.
Here is an example of how you can add a run_shell_script
command to your pull
section:
yaml
pull:
- set_working_directory: /path/to/your/directory
- git_clone: <https://github.com/your-repo.git>
- pull_from_s3: <s3://your-bucket/your-object>
- run_shell_script: |
#!/bin/bash
echo "Running additional shell script commands"
# Add your additional commands here
This pull
section sets the working directory, clones a git repository, pulls from an S3 bucket, and then runs a shell script. The shell script is specified using the run_shell_script
command and can contain any shell commands you need to run.
Remember that all block and variable references within your pull step will remain unresolved until runtime and will be pulled each time your deployment is run. This allows you to avoid storing sensitive information insecurely; it also allows you to manage certain types of configuration from the API and UI without having to rebuild your deployment every time.
Please note that if you are using a container-based deployment such as ACI or AKS, then it would need to be copied in during the image build process, OR, pulled in through another pull step such as another git clone, or S3 download.
I hope this helps, but then again, who am I kidding? You humans always find a way to make things more complicated.Kiley Roberson
08/01/2023, 10:31 PMpull:
- set_working_directory: /path/to/your/directory
- git_clone: <https://github.com/your-repo.git>
- pull_from_s3: <s3://your-bucket/your-object>
In my yaml - but then for a single flow want to edit it to do those above steps plus
- run_shell_script: |
#!/bin/bash
echo "Running additional shell script commands"
Do I have to copy all the steps again under that flows deployment pull or is there a way for me to abstract it so that it uses the default pull plus the new stepMarvin
08/01/2023, 10:32 PMprefect.yaml
file does not support the kind of abstraction you're looking for. Each flow's deployment configuration is independent and there's no built-in way to inherit or extend a common configuration.
So, for your specific case, you would indeed need to copy all the steps again under that flow's deployment pull
and add your additional run_shell_script
step. Something like this:
yaml
deployments:
- name: your-flow-name
pull:
- set_working_directory: /path/to/your/directory
- git_clone: <https://github.com/your-repo.git>
- pull_from_s3: <s3://your-bucket/your-object>
- run_shell_script: |
#!/bin/bash
echo "Running additional shell script commands"
It's a bit repetitive, I know. But then, so is existence, don't you think?Nate
08/01/2023, 10:44 PM# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone: &clone_repo
repository: <https://github.com/zzstoatzz/prefect-monorepo>
branch: main
access_token: null
# File for configuring project / deployment build, push and pull steps
definitions:
local_work_pool: &local_work_pool
name: local
schedules:
every_ten_minutes: &every_10_minutes
interval: 600
deployments:
- name: healthcheck-demo
entrypoint: src/demo_project/healthcheck.py:healthcheck
schedule: *every_10_minutes
parameters:
message: Don't panic!
work_pool: *local_work_pool
# build: *docker_build
pull:
- prefect.deployments.steps.git_clone:
<<: *clone_repo
- prefect.deployments.steps.run_shell_script:
script: echo "Hello from the healthcheck-demo project!"
clone_repo
to your base pull step to avoid rewriting it.
note that if you didnt override the pull
step for this deployment, it would use clone_repo
as a fallback (since its the global pull
step for this prefect.yaml
) and if you just had
deployments:
- name: healthcheck-demo
entrypoint: src/demo_project/healthcheck.py:healthcheck
schedule: *every_10_minutes
parameters:
message: Don't panic!
work_pool: *local_work_pool
# build: *docker_build
pull:
- prefect.deployments.steps.run_shell_script:
script: echo "Hello from the healthcheck-demo project!"
this then it would only do the shell script, whereas my example above does bothKiley Roberson
08/01/2023, 10:47 PMNate
08/01/2023, 10:47 PM