https://prefect.io logo
Title
h

Henning Holgersen

10/16/2022, 2:02 PM
I have pondered deployment patterns in Prefect 2 a little more, and come up with something I might like: https://github.com/radbrt/orion_flows Hopefully the readme is understandable, but the gist of it is to create deployments in Python files (the
Deployment
class), in separate files that imports the flow and creates+applies a deployment. These files have a common naming pattern, all ending with
.deployment.py
, so that github action can pick them up and run them. This way, the flow file itself remains clean, and we don’t hardcode any config in the github action yaml, making it fully configurable. Thanks to @Emil Christensen for explaining the point of deployments to me in a way I finally understood. Oh, and there is also a cookiecutter template to create a new skeleton flow.
🙌 10
:marvin: 5
👀 1
:thank-you: 2
e

Emil Christensen

10/17/2022, 12:15 PM
@Henning Holgersen This is amazing, and so well documented 🙌 Beautiful work!
:thank-you: 1
r

redsquare

10/17/2022, 10:55 PM
@Henning Holgersen useful thanks, quick question - what does KubernetesJob.load("simple") do, I cant find it documented anywhere
h

Henning Holgersen

10/18/2022, 5:17 AM
Good question, and I’d love for someone like @Anna Geller to chime in on the details. Converting from a command-line approach to the Deployment class, I had some trouble getting the deployment to use my own image. It seemed like the infra override didn’t work. Adding a kubernetesjob block fixed that - but the block does not specify much other than the image name. And because the kubernetesjob doesn’t have a config for namespace, I still need to use infra override. I have thought about subclassing the Deployment for our org, just to provide some common defaults for our setup.
a

Anna Geller

10/18/2022, 1:41 PM
Totally understand why someone may like Python deployments but I’m slightly biased in favor of CLI exactly for the reasons mentioned (can't easily pass image built in CI or Git commit hash) and because you explicitly declare how deployments should look like. I’ve used subprocess command in this repo to get the best of both worlds (easy looping, variables and block definition in Python but running CLI commands still) - check it out if you’re interested in marrying both worlds together (scripts that start with numbers) -- note that those scripts here are meant to be run locally, not optimized for CI/CD yet There is no right or wrong approach, that's why we have both options available
r

redsquare

10/18/2022, 2:18 PM
@Anna Geller yeah I like the ability to automate things like required secrets from the flow to be added to the customizations of the job and schedules etc
h

Henning Holgersen

10/18/2022, 2:25 PM
Wow… that subprocess command got extensive, but I see the appeal and it can of course be abstracted away to an even better deployment function - best of both worlds plus the brevity and sensible defaults that casual developers need.
a

Anna Geller

10/18/2022, 5:31 PM
Yes exactly, hard to give any single recommendation or golden path because everyone may want to deploy their flows a little differently
Our general preference is that the product itself is unopinionated and supports various patterns, and we can provide more constrained recipes supporting specific opinionated approaches
💯 2
s

Serina

10/19/2022, 5:04 PM
Hi @Henning Holgersen, this looks great! What do you think about adding this to the recipes-repo README? I can also link to it unless you’d like to contribute a PR yourself 🙂
h

Henning Holgersen

10/19/2022, 5:12 PM
Hi @Serina, happy to hear it looks useful. Feel free to add it to the recipes repo, it is probably faster for both of us if you add the link yourself - I don’t count PRs.
👍 1
🎉 1
💙 1
:thank-you: 1
r

redsquare

10/25/2022, 1:31 PM
@Henning Holgersen I found your project super helpful, I have built on top of it and have changed the github action to only deploy if there are actual changes to any of the project files (flow | *.deployment.py | requirements.txt), then using the project folder name as the matrix so I can install all deployments for a flow in a single job rather than a job for each deployment
🔥 1
:gratitude-thank-you: 1
name: Register & Deploy Prefect flows
on:
push:
workflow_dispatch:
inputs:
pattern:
description: "Project Name or prohect name Pattern? (defaults to all)"
required: false
default: ".*"
permissions:
contents: read
jobs:
establish-environment-job:
runs-on: ubuntu-latest
outputs:
environment: ${{ steps.get-branch.outputs.environment }}
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
-  name: checkout-repo
uses: actions/checkout@v3
with:
fetch-depth: 0
-  name: get-branch
id: get-branch
run: |
echo "Running on branch ${{ github.ref }}"
if [ "${{ github.ref }}" = "refs/heads/master" ]; then
echo "environment=production" >> $GITHUB_OUTPUT
elif [ "${{ github.ref }}" = "refs/heads/uat" ]; then
echo "environment=uat" >> $GITHUB_OUTPUT
else
echo "environment=test" >> $GITHUB_OUTPUT
fi
-  name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v33
with:
json: 'true'
dir_names: 'true'
files: |
projects/**
-  id: set-matrix
run: |
echo "::set-output name=matrix::${{ steps.changed-files.outputs.all_changed_files }}"
deploy-flows:
needs: establish-environment-job
runs-on: ubuntu-latest
container:
image: prefecthq/prefect:2.6.4-python3.10
strategy:
matrix:
folder: ${{ fromJSON(needs.establish-environment-job.outputs.matrix) }}
max-parallel: 3
environment:
name: ${{ needs.establish-environment-job.outputs.environment }}
env:
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_S3_BLOCK: ${{ secrets.PREFECT_S3_BLOCK }}
PREFECT_WORK_QUEUE: ${{ secrets.PREFECT_WORK_QUEUE }}
PREFECT_K8_JOB_BLOCK: ${{ secrets.PREFECT_K8_JOB_BLOCK }}
steps:
- id: checkout-repo
uses: actions/checkout@v3
- id: get-current-folder
run: |
echo "${{ matrix.folder }}"
- id: get-envrionment
shell: bash
run: echo ${{ needs.establish-environment-job.outputs.environment }}
- id: install-python-dependencies
run: |
pip install s3fs
- id: prefect-cloud-login
run: |
prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_KEY }}
prefect work-queue ls
- id: deploy-flows
run: |
cd ${{ matrix.folder }}
pip install -r requirements.txt
find -name "*.deployment.py" | while read fname; do
python $fname
done
h

Henning Holgersen

10/25/2022, 1:41 PM
Wow, thank you @redsquare ! That addition has been on my todo-list. Awesome to see it.
:gratitude-thank-you: 1
r

redsquare

10/25/2022, 1:43 PM
I have some environment steps in my version so that I can use environments in github and secrets so ignore that
I figured a job per deployment on a large repo would take a while - especially when most would not be needed