Built Docker image not used by my worker (/opt/prefect/my_flows not found by worker, but findable locally in container)
Hi all ! Hope you're doing well ! I'm currently stuck deploying my flows on my GKE cluster. Here is my configuration : I'm using
prefect deploy
from my
my_flows
directory containing my
prefect.yaml
file
name: my_flows
prefect-version: 2.11.3
build:
- prefect.deployments.steps.run_shell_script:
id: get-commit-hash
script: git rev-parse --short HEAD
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
requires: prefect-docker>=0.3.0
image_name: my_repo/my_flows
tag: "{{ get-commit-hash.stdout }}"
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
credentials: "{{ prefect.blocks.docker-registry.docker-registry }}"
pull:
- prefect.projects.steps.set_working_directory:
directory: /opt/prefect/my_flows
deployments:
- name: process-lyrics-deployment
version:
tags: ["process-lyrics"]
description:
schedule: {}
flow_name:
entrypoint: process_lyrics/main.py:process_lyrics
parameters:
work_pool:
name: pool-1
work_queue_name:
job_variables:
image: "{{ image_name }}"
And this dockerfile
FROM prefecthq/prefect:2.11.3-python3.8
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . /opt/prefect/my_flows/
When I check my container locally, I can find everything in the directory
/opt/prefect/my_flows/
(as wanted)
But when I run my job on my cluster I get this error...
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/my_flows'
I'm using one single worker, with the same prefect version as my CLI (
prefect:2.11.3-python3.8
), and configured with
pullPolicy: Always
Since I've seen this forum post :
https://discourse.prefect.io/t/how-to-configure-the-pull-step-for-a-project-building-a-docker-image/2777/2
But it doesn't seem to work for me... Thanks in advance for the help 🙏
