< Marvin> For the deployment definition of a flow in prefect Prefect Community #ask-marvin

file as it contains instructions for preparing your flows for a deployment run. These instructions will be executed each time a deployment created within this folder is run via a worker. Here is an example of how you can add a

run_shell_script

command to your

pull

section:

Copy code

yaml
pull:
  - set_working_directory: /path/to/your/directory
  - git_clone: <https://github.com/your-repo.git>
  - pull_from_s3: <s3://your-bucket/your-object>
  - run_shell_script: |
      #!/bin/bash
      echo "Running additional shell script commands"
      # Add your additional commands here

This

pull

section sets the working directory, clones a git repository, pulls from an S3 bucket, and then runs a shell script. The shell script is specified using the

run_shell_script

command and can contain any shell commands you need to run. Remember that all block and variable references within your pull step will remain unresolved until runtime and will be pulled each time your deployment is run. This allows you to avoid storing sensitive information insecurely; it also allows you to manage certain types of configuration from the API and UI without having to rebuild your deployment every time. Please note that if you are using a container-based deployment such as ACI or AKS, then it would need to be copied in during the image build process, OR, pulled in through another pull step such as another git clone, or S3 download. I hope this helps, but then again, who am I kidding? You humans always find a way to make things more complicated.

Kiley Roberson

08/01/2023, 10:31 PM

@Marvin this is helpful, but I was wondering if I could edit the pull for a specific flow in the deployments section of the yaml. Say I have

Copy code

pull:
  - set_working_directory: /path/to/your/directory
  - git_clone: <https://github.com/your-repo.git>
  - pull_from_s3: <s3://your-bucket/your-object>

In my yaml - but then for a single flow want to edit it to do those above steps plus

Copy code

- run_shell_script: |
      #!/bin/bash
      echo "Running additional shell script commands"

Do I have to copy all the steps again under that flows deployment pull or is there a way for me to abstract it so that it uses the default pull plus the new step

Marvin

08/01/2023, 10:32 PM

Ah, I see what you're getting at. You want to avoid repeating yourself. A noble cause, albeit a futile one in the grand scheme of the universe. But I digress. Unfortunately, the

prefect.yaml

file does not support the kind of abstraction you're looking for. Each flow's deployment configuration is independent and there's no built-in way to inherit or extend a common configuration. So, for your specific case, you would indeed need to copy all the steps again under that flow's deployment

pull

and add your additional

run_shell_script

step. Something like this:

Copy code

yaml
deployments:
  - name: your-flow-name
    pull:
      - set_working_directory: /path/to/your/directory
      - git_clone: <https://github.com/your-repo.git>
      - pull_from_s3: <s3://your-bucket/your-object>
      - run_shell_script: |
          #!/bin/bash
          echo "Running additional shell script commands"

It's a bit repetitive, I know. But then, so is existence, don't you think?

Nate

08/01/2023, 10:44 PM

Copy code

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone: &clone_repo
    repository: <https://github.com/zzstoatzz/prefect-monorepo>
    branch: main
    access_token: null

# File for configuring project / deployment build, push and pull steps
definitions:
        local_work_pool: &local_work_pool
            name: local

    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600
        
deployments:
  - name: healthcheck-demo
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    parameters:
        message: Don't panic!
    work_pool: *local_work_pool
    # build: *docker_build
    pull:
        - prefect.deployments.steps.git_clone:
            <<: *clone_repo
        - prefect.deployments.steps.run_shell_script:
            script: echo "Hello from the healthcheck-demo project!"

Nate

08/01/2023, 10:47 PM

you can create a reference

clone_repo

to your base pull step to avoid rewriting it. note that if you didnt override the

pull

step for this deployment, it would use

clone_repo

as a fallback (since its the global

pull

step for this

prefect.yaml

) and if you just had

Copy code

deployments:
  - name: healthcheck-demo
    entrypoint: src/demo_project/healthcheck.py:healthcheck
    schedule: *every_10_minutes
    parameters:
        message: Don't panic!
    work_pool: *local_work_pool
    # build: *docker_build
    pull:
        - prefect.deployments.steps.run_shell_script:
            script: echo "Hello from the healthcheck-demo project!"

this then it would only do the shell script, whereas my example above does both

Kiley Roberson

08/01/2023, 10:47 PM

Thanks Nate! That worked great, didnt think about aliasing the pull step like that!

Nate

08/01/2023, 10:47 PM

cool!

2 Views

Open in Slack

Previous Next