Not sure if this is the right place to get help but I have installed a self-hosted prefect server and I have also installed a prefect worker however when I try to create a work-pool in the UI it is not giving any option under the

Select the infrastructure you want to use to execute your flow runs

prompt. Anyone can help with some insights or instruction for self hosted prefcet 3 server ?

hi @Hakim - can you share which version of prefect you're running?

Prefect 3.0

sorry, I mean more specifically which exact version you're using

Oh sure! this one

2024.10.17192636'

is that a helm chart version? im looking for something like

3.0.x

how are you running the server?

On kuberntes deployed uisng the helm chart

ok thanks!

I see these labels on the app in Argo

<http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-server
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 2-latest
    <http://helm.sh/chart|helm.sh/chart>: prefect-server-0.0.0
    pod-template-hash: 89c95974
    prefect-version: 2-latest

I've actually reproduced this just with

» docker run -p 4200:4200 -d --rm prefecthq/prefect:3-latest -- prefect server start --host 0.0.0.0

and navigating to

localhost:4200

Oh okay I apprecciate it

ok good news! this is something that is already fixed in

main

, and we will cut a release later today or tomorrow so

3-latest

should do the right thing after thats out in the meantime though,

3.0.4

does not have this problem so in the

values.yaml

for your server deployment, you can select the image tag and this one should work for you

prefecthq/prefect:3.0.4-python3.12

Nice!!! I appreciate the insane turn around time! Thank you so much Nate. I will make these adjustments and be on the lookout for the new released version.

Okay that worked! Now I was hoping to see if you guys have a way to help me setup a deployment for my sample flow with GitLab CI that has EKS environments linked to it I only see github-actions in the guide and it also lacks the explanation on how to link that to the worker which is running in my EKS cluster

in general this is the pattern i'd follow (docs) • create a work pool that represents your cluster as described in the kubernetes guide • define your deployments in python (

flow.from_source(source="<https://mygitlaburletc>", entrypoint=...).deploy()

) or in a `prefect.yaml` ◦ using either

from_source

(python) or a

pull

step (yaml) you give the deployment the info about where the code lives (gitlab) ◦ using

.deploy()

or the

work_pool

section of the yaml, you say which work pool the deployment should get its infra config from (your new k8s work pool) • deploy flows when you push code to gitlab ◦ set

PREFECT_API_URL

in the env of your gitlab CI ◦ run

prefect --no-prompt deploy --all

or

python deploy.py

depending on whether you define deployments in yaml or python

Nice! Okay this looks good. I will try to get these steps going to see where that lands me. I think I am going to go with the prefect.yaml approach combined with the workpool. So I believe that's the `prefect --no-prompt deploy --all`route. I guess I can have a deploy stage which will invoke that

prefect.yaml

file through that command giben they are in the same root level directory right ?

yes! by default we assume that

prefect.yaml

is at your repo root, but you can pass

--prefect-file some/path/prefect.yaml

if you end up wanting to break things out

» prefect deploy --help

 Usage: prefect deploy [OPTIONS] [ENTRYPOINT]

 Create a deployment to deploy a flow from this project.
 Should be run from a project root directory.
...

--prefect-file                       PATH     Specify a custom path to a prefect.yaml file [default: prefect.yaml]

here is an example

prefect.yaml

with some valid deployment definitions, and the CI (granted for GHA, not gitlab, but the bash should be the same)

Do the images get pushed to a registry (we use ECR) or are they just passed to the deploy step through reference `image: "{{ build-image.image }}"`Do they need to ?

its up to you! you can define image build/push as a thing that should happen during `prefect deploy` or instead run

docker build

/

docker push

etc yourself elsewhere and then hardcode or template the resulting image tag so your deployment knows to use it the

"{{ build-image.image }}"

syntax is just a special case of templating where you opt to build your image as a part of

prefect deploy

and the `build-image` step produces an image that you can use to define the

image

on your deployment's

job_variables

For now to get this of the ground and into the server I am just gonna use the reference method that we have as part of the template. Okay I think this good to get me started with this step. Hey Nate you have been great! Thank you for all your help with this I really appreciate it!

sure thing!

Hi @Nate not sure if you would have any idea about this but here is my

prefect.yaml

to get ran during CI deploy job

# Generic metadata about this project
name: flows
prefect-version: 3.0.0



# build section allows you to manage and build docker images
build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.6.1
      image_name: "lab-prefect-3"
      tag: latest
      dockerfile: Dockerfile
      platform: "linux/amd64"

## push section allows you to manage if and how this project is uploaded to remote locations
push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.6.1
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

# the definitions section allows you to define reusable components for your deployments
definitions:
  tags: &common_tags
    - "eks"
  work_pool: &common_work_pool
    name: "my-pool"
    job_variables:
      image: "{{ build-image.image }}"

# the deployments section allows you to provide configuration for deploying flows
deployments:
  - name: "lab-prefect-3-deployment"
    tags: *common_tags
    schedule: null
    entrypoint: "src/flows/flow.py:hello"
    parameters:
      name: "Hakim"
    work_pool: *common_work_pool

But I think during the build phase it is failing with this error

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pydantic/json_schema.py", line 337, in build_schema_type_to_method
    mapping[key] = getattr(self, method_name)
AttributeError: 'GenerateEmptySchemaForUserClasses' object has no attribute 'invalid_schema'

the step itself accepts references to

credentials

one useful insight is that all those steps are just fully qualified function names

prefect_docker.deployments.steps.push_docker_image

etc https://github.com/PrefectHQ/prefect/blob/ca9e7bdcc947c6cb9ed42e0a0c9ed45c503c816c[…]integrations/prefect-docker/prefect_docker/deployments/steps.py

Got it thanks. But where can I define the value for this

prefect.blocks.docker-registry-credentials.dev-registry

you can go in the UI and create a block of that type called

dev-registry

but you dont need to create a block, you can just auth with docker or your registry otherwise before calling

prefect deploy

as shown in the docs

Got it so I can just authenticate to the ecr registry and the build step will push to that registry ?

yeah based on your value of

image

which contains the ref to the registry repo etc

Ah I understand so in the image name I would provide the full name and path to the ECR registry

Nice Yeah I think I get it now. Thank you @Nate! always a huge help!