https://prefect.io logo
Title
e

Emma Rizzi

04/13/2023, 8:58 AM
hello, I struggle with the new projects feature and couldn't find any detailed example for dockerized deployments, nor any list of parameters needed/accepted in prefect.yaml and deployment.yaml. I have deployed a flow with
prefect deployment build
and I am now trying to migrate to projects to create flows as docker images, but
prefect deploy
fails, details in thread
1
prefect.yaml
name: flow
prefect-version: 2.10.1

build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: auto
    push: True
    credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"

pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect/flow
deployment.yaml
# base metadata
name: hello-world-deploy
version: null
tags: []
description: null
schedule: null

# flow-specific fields
flow_name: check
entrypoint: flow.py:check
path: null
parameters: {}
parameter_openapi_schema: null

# infra-specific fields
work_pool:
  name: default-agent-pool
  work_queue_name: kubernetes
  job_variables: {}
Dockerfile
FROM xxx-custom-image/prefect:base-3.10-0.1

COPY flow/requirements.txt requirements.txt
RUN pip install -r requirements.txt
All files in the same directory, my flow was built and ran successfully with the following command :
prefect deployment build flow.py:check -n k8sjob -sb s3-bucket/s3 -ib kubernetes-job/eouser-job -a --path test/ --override image=xxx/prefect:xarray-0.1 -q kubernetes
But
prefect deploy
returns error :
'NoneType' object has no attribute 'split'
a

alex

04/13/2023, 1:16 PM
Hey @Emma Rizzi! There’s a known bug where the flow name and entrypoint aren’t pulled from the
deployment.yaml
file when running
prefect deploy
. This will be fixed in the next release, but in the meantime you can pass either the flow name or entrypoint via the CLI like this
prefect deploy -f check
for flow name or this
prefect deploy flow.py:check
for entrypoint. Note that the deploy CLI will error if both flow name and entrypoint as provided. I also noticed that your deployment is set to schedule flow runs in the
default-agent-pool
. Project can only deploy to typed work pools, so you will need to create a typed work pool to use when deploying this project.
e

Emma Rizzi

04/13/2023, 1:49 PM
@Alex Beatson thanks for looking into this, I tried your suggestion, it said to register the flow first then I had a file not found error, I tried to provide the absolute path also but same error, any idea ?
a

alex

04/13/2023, 1:52 PM
Hmm, it looks like the location of your flow is missing the
~
. Can you send the contents of your
.prefect/flow.json
file? The
.prefect
directory should be in your
flow-docker
directory.
1
e

Emma Rizzi

04/13/2023, 2:03 PM
That was it ! I don't have any
.prefect
directory here, seems that didn't configure it right with project init, I ran again
prefect project init --recipe docker
and it started building the image! New error tho,
stat flow/requirements.txt: file does not exist
, I want to include other files in my flow so I used docker
COPY
, seems I missed something in context configuration ?
a

alex

04/13/2023, 2:07 PM
Based on your directory structure I think you can update your Dockerfile to this to resolve that error:
FROM xxx-custom-image/prefect:base-3.10-0.1

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
You’ll also want to update your build step to reference your custom Dockerfile:
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: Dockerfile
    push: True
    credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"
If you pass
auto
for
dockerfile
then prefect tries to automatically create a Dockerfile on your behalf.
e

Emma Rizzi

04/13/2023, 2:21 PM
I feel so dumb for missing that, spent too much time in the new prefect doc i forgot how to docker 🥲 almost there... it seems my credentials are not loaded right, i followed the last example here: https://prefecthq.github.io/prefect-docker/projects/steps/#prefect_docker.projects.steps.BuildDockerImageResult But i get
username=credentials.get("username"),
AttributeError: 'str' object has no attribute 'get'
My block screened here
a

alex

04/13/2023, 2:25 PM
Ah, looks like there’s a type in that example. Blocks are referenced with
prefect.blocks
. Here’s an updated build action:
build:
-prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: Dockerfile
    push: True
    credentials: "{{ prefect.blocks.docker-registry-credentials.docker-hub }}"
e

Emma Rizzi

04/13/2023, 2:30 PM
Thanks it finished! just need to deploy my worker and all good! If I may ask one last request, is it possible to add the infrastructure-block parameter in deployment or prefect yaml ?
a

alex

04/13/2023, 2:32 PM
Projects work with typed work pool and workers which are a replacement for infrastructure blocks. Which infrastructure block did you want to use for this deployment?
e

Emma Rizzi

04/13/2023, 2:33 PM
i use a kubernetes job infrastructure with a custom job template
a

alex

04/13/2023, 2:35 PM
Cool, you can use a
kubernetes
typed work pool with the Kubernetes worker. The Kubernetes work very similarly to the
KubernetesJob
block, but the configuration is handled by the work pool instead of a block. The custom job manifest can be set on the work pool and the way to do that is documented here.
e

Emma Rizzi

04/13/2023, 2:37 PM
ah super! I'll dig into this, thanks a lot for your support !! prefect team is always amazing :catjam:
:catjam: 1