https://prefect.io logo
Powered by
Title
e

Emma Rizzi

04/13/2023, 8:58 AM
hello, I struggle with the new projects feature and couldn't find any detailed example for dockerized deployments, nor any list of parameters needed/accepted in prefect.yaml and deployment.yaml. I have deployed a flow with 
prefect deployment build
and I am now trying to migrate to projects to create flows as docker images, but 
prefect deploy
fails, details in thread
✅ 1
prefect.yaml 
name: flow
prefect-version: 2.10.1

build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: auto
    push: True
    credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"

pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect/flow
deployment.yaml 
# base metadata
name: hello-world-deploy
version: null
tags: []
description: null
schedule: null

# flow-specific fields
flow_name: check
entrypoint: flow.py:check
path: null
parameters: {}
parameter_openapi_schema: null

# infra-specific fields
work_pool:
  name: default-agent-pool
  work_queue_name: kubernetes
  job_variables: {}
Dockerfile 
FROM xxx-custom-image/prefect:base-3.10-0.1

COPY flow/requirements.txt requirements.txt
RUN pip install -r requirements.txt
All files in the same directory, my flow was built and ran successfully with the following command : 
prefect deployment build flow.py:check -n k8sjob -sb s3-bucket/s3 -ib kubernetes-job/eouser-job -a --path test/ --override image=xxx/prefect:xarray-0.1 -q kubernetes
But 
prefect deploy
returns error : 
'NoneType' object has no attribute 'split'
a

alex

04/13/2023, 1:16 PM
Hey @Emma Rizzi! There’s a known bug where the flow name and entrypoint aren’t pulled from the 
deployment.yaml
file when running 
prefect deploy
. This will be fixed in the next release, but in the meantime you can pass either the flow name or entrypoint via the CLI like this 
prefect deploy -f check
for flow name or this 
prefect deploy flow.py:check
for entrypoint. Note that the deploy CLI will error if both flow name and entrypoint as provided. I also noticed that your deployment is set to schedule flow runs in the 
default-agent-pool
. Project can only deploy to typed work pools, so you will need to create a typed work pool to use when deploying this project.
e

Emma Rizzi

04/13/2023, 1:49 PM
@Alex Beatson thanks for looking into this, I tried your suggestion, it said to register the flow first then I had a file not found error, I tried to provide the absolute path also but same error, any idea ?
a

alex

04/13/2023, 1:52 PM
Hmm, it looks like the location of your flow is missing the 
~
. Can you send the contents of your 
.prefect/flow.json
file? The 
.prefect
directory should be in your 
flow-docker
directory.
✅ 1
e

Emma Rizzi

04/13/2023, 2:03 PM
That was it ! I don't have any 
.prefect
directory here, seems that didn't configure it right with project init, I ran again 
prefect project init --recipe docker
and it started building the image! New error tho, 
stat flow/requirements.txt: file does not exist
, I want to include other files in my flow so I used docker 
COPY
, seems I missed something in context configuration ?
a

alex

04/13/2023, 2:07 PM
Based on your directory structure I think you can update your Dockerfile to this to resolve that error: 
FROM xxx-custom-image/prefect:base-3.10-0.1

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
You’ll also want to update your build step to reference your custom Dockerfile: 
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: Dockerfile
    push: True
    credentials: "{{ prefect.block.docker-registry-credentials.docker-hub }}"
If you pass 
auto
for 
dockerfile
then prefect tries to automatically create a Dockerfile on your behalf.
e

Emma Rizzi

04/13/2023, 2:21 PM
I feel so dumb for missing that, spent too much time in the new prefect doc i forgot how to docker 🥲 almost there... it seems my credentials are not loaded right, i followed the last example here: https://prefecthq.github.io/prefect-docker/projects/steps/#prefect_docker.projects.steps.BuildDockerImageResult But i get 
username=credentials.get("username"),
AttributeError: 'str' object has no attribute 'get'
My block screened here
a

alex

04/13/2023, 2:25 PM
Ah, looks like there’s a type in that example. Blocks are referenced with 
prefect.blocks
. Here’s an updated build action: 
build:
-prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: xxx/orion
    tag: test-1.0
    dockerfile: Dockerfile
    push: True
    credentials: "{{ prefect.blocks.docker-registry-credentials.docker-hub }}"
e

Emma Rizzi

04/13/2023, 2:30 PM
Thanks it finished! just need to deploy my worker and all good! If I may ask one last request, is it possible to add the infrastructure-block parameter in deployment or prefect yaml ?
a

alex

04/13/2023, 2:32 PM
Projects work with typed work pool and workers which are a replacement for infrastructure blocks. Which infrastructure block did you want to use for this deployment?
e

Emma Rizzi

04/13/2023, 2:33 PM
i use a kubernetes job infrastructure with a custom job template
a

alex

04/13/2023, 2:35 PM
Cool, you can use a 
kubernetes
typed work pool with the Kubernetes worker. The Kubernetes work very similarly to the 
KubernetesJob
block, but the configuration is handled by the work pool instead of a block. The custom job manifest can be set on the work pool and the way to do that is documented here.
e

Emma Rizzi

04/13/2023, 2:37 PM
ah super! I'll dig into this, thanks a lot for your support !! prefect team is always amazing :catjam:
:catjam: 1
3 Views
#prefect-communityJoin Slack
Powered by