Chris
04/01/2024, 6:06 PMjobs:
build:
name: Deploy Prefect with Modal Work Pool
runs-on: ubuntu-latest
environment: Development
env:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
steps:
- uses: actions/checkout@v4
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Prefect Auth
uses: PrefectHQ/actions-prefect-auth@v1
with:
prefect-api-key: ${{ secrets.PREFECT_API_KEY }}
prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
- name: Create Work Pool
run: echo "y" | prefect work-pool create --type modal:push --provision-infra modal-pool || true
- name: Prefect Deploy
run: prefect deploy
and a prefect.yaml of
pull:
- prefect.deployments.steps.git_clone:
repository: redacted
branch: feature/prefect-modal-testing
access_token: "{{ secrets.GITHUB_TOKEN }}"
build:
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker
image_name: arnold/base
tag: dev
dockerfile: dockerfiles/Dockerfile.service
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"
additional_tags: "{{ build-image.additional_tags }}"
deployments:
- name: test_model
entrypoint: modal_apps/modal_prefect_test.py:run_app
When do the pull, build, push steps happen here? I don't see any logs associated with these stepsKevin Grismore
04/01/2024, 6:32 PMprefect deploy
and pull steps run right before the start of a flow runKevin Grismore
04/01/2024, 6:33 PMrun: prefect deploy
exactly? You'll need to specify deployments by name like prefect deploy -n test_model
or do prefect deploy --all
to run the steps for all the deployments in your prefect.yaml
Chris
04/01/2024, 7:01 PMChris
04/01/2024, 7:06 PMKevin Grismore
04/01/2024, 7:06 PMChris
04/01/2024, 7:36 PMChris
04/01/2024, 7:37 PMKevin Grismore
04/01/2024, 7:38 PMwork_pool
sectionKevin Grismore
04/01/2024, 7:38 PMjob_variables
which is a dictionary where on of the keys is image
Chris
04/01/2024, 7:38 PMpull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/essexlabs/arnold.git>
branch: feature/prefect-modal-testing
access_token: "{{ secrets.GITHUB_TOKEN }}"
build:
- prefect_docker.deployments.steps.build_docker_image:
requires: prefect-docker
image_name: arnold/base
tag: dev
dockerfile: dockerfiles/Dockerfile.deps
push: # Prefect blocks add docker stuff
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"
additional_tags: "{{ build-image.additional_tags }}"
deployments:
- name: test_model
entrypoint: modal_apps/test_modal_app.py:run_app
work_pool:
name: modal-pool
I have this nowChris
04/01/2024, 7:39 PMChris
04/01/2024, 7:40 PMKevin Grismore
04/01/2024, 7:43 PMid
so you can refer to the outputs of previous steps by name
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build-image <-- now "{{ build-image.<output-name> }}" will work in later steps
requires: prefect-docker
image_name: arnold/base
tag: dev
dockerfile: dockerfiles/Dockerfile.deps
then on your deployment, you can access the combined image:tag
string as "{{ build-image.image }}"
work_pool:
name: modal-pool
job_variables:
image: "{{ build-image.image }}"
Kevin Grismore
04/01/2024, 7:44 PMChris
04/01/2024, 7:45 PMKevin Grismore
04/01/2024, 7:46 PMChris
04/01/2024, 7:46 PM