https://prefect.io logo
Channels
Powered by
k

Kiley Roberson

08/01/2023, 10:27 PM
@Marvin For the deployment definition of a flow in prefect.yaml, can I specify a unique pull step that uses the default 
pull
defined in the yaml but also adds an additional step (For example runs a run_shell_script command)? And if so what would the syntax for that be?
m

Marvin

08/01/2023, 10:28 PM
Yes, you can specify a unique pull step in your 
prefect.yaml
that uses the default 
pull
defined in the yaml and also adds an additional step. The pull section is the most important section within the 
prefect.yaml
file as it contains instructions for preparing your flows for a deployment run. These instructions will be executed each time a deployment created within this folder is run via a worker. Here is an example of how you can add a 
run_shell_script
command to your 
pull
section:
Copy code
yaml
pull:
  - set_working_directory: /path/to/your/directory
  - git_clone: <https://github.com/your-repo.git>
  - pull_from_s3: <s3://your-bucket/your-object>
  - run_shell_script: |
      #!/bin/bash
      echo "Running additional shell script commands"
      # Add your additional commands here
This 
pull
section sets the working directory, clones a git repository, pulls from an S3 bucket, and then runs a shell script. The shell script is specified using the 
run_shell_script
command and can contain any shell commands you need to run. Remember that all block and variable references within your pull step will remain unresolved until runtime and will be pulled each time your deployment is run. This allows you to avoid storing sensitive information insecurely; it also allows you to manage certain types of configuration from the API and UI without having to rebuild your deployment every time. Please note that if you are using a container-based deployment such as ACI or AKS, then it would need to be copied in during the image build process, OR, pulled in through another pull step such as another git clone, or S3 download. I hope this helps, but then again, who am I kidding? You humans always find a way to make things more complicated.
k

Kiley Roberson

08/01/2023, 10:31 PM
@Marvin this is helpful, but I was wondering if I could edit the pull for a specific flow in the deployments section of the yaml. Say I have
Copy code
pull:
  - set_working_directory: /path/to/your/directory
  - git_clone: <https://github.com/your-repo.git>
  - pull_from_s3: <s3://your-bucket/your-object>
In my yaml - but then for a single flow want to edit it to do those above steps plus
Copy code
- run_shell_script: |
      #!/bin/bash
      echo "Running additional shell script commands"
Do I have to copy all the steps again under that flows deployment pull or is there a way for me to abstract it so that it uses the default pull plus the new step
m

Marvin

08/01/2023, 10:32 PM
Ah, I see what you're getting at. You want to avoid repeating yourself. A noble cause, albeit a futile one in the grand scheme of the universe. But I digress. Unfortunately, the 
prefect.yaml
file does not support the kind of abstraction you're looking for. Each flow's deployment configuration is independent and there's no built-in way to inherit or extend a common configuration. So, for your specific case, you would indeed need to copy all the steps again under that flow's deployment 
pull
and add your additional 
run_shell_script
step. Something like this:
Copy code
yaml
deployments:
  - name: your-flow-name
    pull:
      - set_working_directory: /path/to/your/directory
      - git_clone: <https://github.com/your-repo.git>
      - pull_from_s3: <s3://your-bucket/your-object>
      - run_shell_script: |
          #!/bin/bash
          echo "Running additional shell script commands"
It's a bit repetitive, I know. But then, so is existence, don't you think?
n

Nate

08/01/2023, 10:44 PM
Copy code
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone: &clone_repo
    repository: <https://github.com/zzstoatzz/prefect-monorepo>
    branch: main
    access_token: null

# File for configuring project / deployment build, push and pull steps
definitions:
        local_work_pool: &local_work_pool
            name: local

    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600
        
deployments:
  - name: healthcheck-demo
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    parameters:
        message: Don't panic!
    work_pool: *local_work_pool
    # build: *docker_build
    pull:
        - prefect.deployments.steps.git_clone:
            <<: *clone_repo
        - prefect.deployments.steps.run_shell_script:
            script: echo "Hello from the healthcheck-demo project!"
you can create a reference 
clone_repo
to your base pull step to avoid rewriting it. note that if you didnt override the 
pull
step for this deployment, it would use 
clone_repo
as a fallback (since its the global 
pull
step for this 
prefect.yaml
) and if you just had
Copy code
deployments:
  - name: healthcheck-demo
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    parameters:
        message: Don't panic!
    work_pool: *local_work_pool
    # build: *docker_build
    pull:
        - prefect.deployments.steps.run_shell_script:
            script: echo "Hello from the healthcheck-demo project!"
this then it would only do the shell script, whereas my example above does both
k

Kiley Roberson

08/01/2023, 10:47 PM
Thanks Nate! That worked great, didnt think about aliasing the pull step like that!
n

Nate

08/01/2023, 10:47 PM
cool!
Powered by