Emma Rizzi
04/13/2023, 8:58 AMprefect deployment build
and I am now trying to migrate to projects to create flows as docker images, but prefect deploy
fails, details in threadname: flow
prefect-version: 2.10.1
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: xxx/orion
tag: test-1.0
dockerfile: auto
push: True
credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"
pull:
- prefect.projects.steps.set_working_directory:
directory: /opt/prefect/flow
# base metadata
name: hello-world-deploy
version: null
tags: []
description: null
schedule: null
# flow-specific fields
flow_name: check
entrypoint: flow.py:check
path: null
parameters: {}
parameter_openapi_schema: null
# infra-specific fields
work_pool:
name: default-agent-pool
work_queue_name: kubernetes
job_variables: {}
FROM xxx-custom-image/prefect:base-3.10-0.1
COPY flow/requirements.txt requirements.txt
RUN pip install -r requirements.txt
prefect deployment build flow.py:check -n k8sjob -sb s3-bucket/s3 -ib kubernetes-job/eouser-job -a --path test/ --override image=xxx/prefect:xarray-0.1 -q kubernetes
But prefect deploy
returns error :
'NoneType' object has no attribute 'split'
alex
04/13/2023, 1:16 PMdeployment.yaml
file when running prefect deploy
. This will be fixed in the next release, but in the meantime you can pass either the flow name or entrypoint via the CLI like this prefect deploy -f check
for flow name or this prefect deploy flow.py:check
for entrypoint. Note that the deploy CLI will error if both flow name and entrypoint as provided.
I also noticed that your deployment is set to schedule flow runs in the default-agent-pool
. Project can only deploy to typed work pools, so you will need to create a typed work pool to use when deploying this project.Emma Rizzi
04/13/2023, 1:49 PMalex
04/13/2023, 1:52 PM~
. Can you send the contents of your .prefect/flow.json
file? The .prefect
directory should be in your flow-docker
directory.Emma Rizzi
04/13/2023, 2:03 PM.prefect
directory here, seems that didn't configure it right with project init, I ran again prefect project init --recipe docker
and it started building the image!
New error tho, stat flow/requirements.txt: file does not exist
, I want to include other files in my flow so I used docker COPY
, seems I missed something in context configuration ?alex
04/13/2023, 2:07 PMFROM xxx-custom-image/prefect:base-3.10-0.1
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
You’ll also want to update your build step to reference your custom Dockerfile:
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: xxx/orion
tag: test-1.0
dockerfile: Dockerfile
push: True
credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"
If you pass auto
for dockerfile
then prefect tries to automatically create a Dockerfile on your behalf.Emma Rizzi
04/13/2023, 2:21 PMusername=credentials.get("username"),
AttributeError: 'str' object has no attribute 'get'
My block screened herealex
04/13/2023, 2:25 PMprefect.blocks
. Here’s an updated build action:
build:
-prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: xxx/orion
tag: test-1.0
dockerfile: Dockerfile
push: True
credentials: "{{ prefect.blocks.docker-registry-credentials.docker-hub }}"
Emma Rizzi
04/13/2023, 2:30 PMalex
04/13/2023, 2:32 PMEmma Rizzi
04/13/2023, 2:33 PMalex
04/13/2023, 2:35 PMkubernetes
typed work pool with the Kubernetes worker. The Kubernetes work very similarly to the KubernetesJob
block, but the configuration is handled by the work pool instead of a block. The custom job manifest can be set on the work pool and the way to do that is documented here.Emma Rizzi
04/13/2023, 2:37 PM