Henning Holgersen
10/16/2022, 2:02 PMDeployment
class), in separate files that imports the flow and creates+applies a deployment. These files have a common naming pattern, all ending with .deployment.py
, so that github action can pick them up and run them. This way, the flow file itself remains clean, and we don’t hardcode any config in the github action yaml, making it fully configurable.
Thanks to @Emil Christensen for explaining the point of deployments to me in a way I finally understood.
Oh, and there is also a cookiecutter template to create a new skeleton flow.Emil Christensen
10/17/2022, 12:15 PMredsquare
10/17/2022, 10:55 PMHenning Holgersen
10/18/2022, 5:17 AMAnna Geller
10/18/2022, 1:41 PMredsquare
10/18/2022, 2:18 PMHenning Holgersen
10/18/2022, 2:25 PMAnna Geller
10/18/2022, 5:31 PMSerina
10/19/2022, 5:04 PMHenning Holgersen
10/19/2022, 5:12 PMredsquare
10/25/2022, 1:31 PMname: Register & Deploy Prefect flows
on:
push:
workflow_dispatch:
inputs:
pattern:
description: "Project Name or prohect name Pattern? (defaults to all)"
required: false
default: ".*"
permissions:
contents: read
jobs:
establish-environment-job:
runs-on: ubuntu-latest
outputs:
environment: ${{ steps.get-branch.outputs.environment }}
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: checkout-repo
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: get-branch
id: get-branch
run: |
echo "Running on branch ${{ github.ref }}"
if [ "${{ github.ref }}" = "refs/heads/master" ]; then
echo "environment=production" >> $GITHUB_OUTPUT
elif [ "${{ github.ref }}" = "refs/heads/uat" ]; then
echo "environment=uat" >> $GITHUB_OUTPUT
else
echo "environment=test" >> $GITHUB_OUTPUT
fi
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v33
with:
json: 'true'
dir_names: 'true'
files: |
projects/**
- id: set-matrix
run: |
echo "::set-output name=matrix::${{ steps.changed-files.outputs.all_changed_files }}"
deploy-flows:
needs: establish-environment-job
runs-on: ubuntu-latest
container:
image: prefecthq/prefect:2.6.4-python3.10
strategy:
matrix:
folder: ${{ fromJSON(needs.establish-environment-job.outputs.matrix) }}
max-parallel: 3
environment:
name: ${{ needs.establish-environment-job.outputs.environment }}
env:
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_S3_BLOCK: ${{ secrets.PREFECT_S3_BLOCK }}
PREFECT_WORK_QUEUE: ${{ secrets.PREFECT_WORK_QUEUE }}
PREFECT_K8_JOB_BLOCK: ${{ secrets.PREFECT_K8_JOB_BLOCK }}
steps:
- id: checkout-repo
uses: actions/checkout@v3
- id: get-current-folder
run: |
echo "${{ matrix.folder }}"
- id: get-envrionment
shell: bash
run: echo ${{ needs.establish-environment-job.outputs.environment }}
- id: install-python-dependencies
run: |
pip install s3fs
- id: prefect-cloud-login
run: |
prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_KEY }}
prefect work-queue ls
- id: deploy-flows
run: |
cd ${{ matrix.folder }}
pip install -r requirements.txt
find -name "*.deployment.py" | while read fname; do
python $fname
done
Henning Holgersen
10/25/2022, 1:41 PMredsquare
10/25/2022, 1:43 PM