https://prefect.io logo
m

morten

07/24/2023, 9:01 AM
Hi all 🙂 Wanted to show how I deployed a worker in a docker environment while using Prefect Cloud. I am somewhat new to docker, so maybe someone can improve it, but I want to describe the challenges I face. What I wanted: Start a docker container which, upon initialization, logs in to my cloud account and automatically starts a Prefect worker. The problem: Prefect already has a docker image, but simply writing a Dockerfile like
Copy code
FROM prefecthq/prefect:2-python3.11 

ARG PREFECT_TK

RUN prefect cloud login --key "${PREFECT_TK}" --workspace "mortenginnerupandreasenzohocom/homelab"

CMD prefect worker start --pool my-process-pool
does not work. The
CMD
line overwrites all the lines in the original prefect image, rendering it all useless. Appending the
CMD
to the
RUN
line with and
&&
does not work, as
RUN
is executed when the image is build, not deployed. Solution: Use multi stage builds. My Dockerfile looks likes this
Copy code
# Install he prefect image and login.
FROM prefecthq/prefect:2-python3.11 AS builder

ARG PREFECT_TK

RUN prefect cloud login --key "${PREFECT_TK}" --workspace "mortenginnerupandreasenzohocom/homelab"

# Cant start worker from first stage, as CMD will override all CMD's in the original prefect image.
# Therefore, a second stage is added, where a CMD can be freely added without consequences for previous
# CMD's.
FROM builder AS starter

CMD prefect worker start --pool my-process-pool
and my compose.yaml looks like this:
Copy code
services:
  worker:
    build:
      context: .
      args:
        PREFECT_TK: ${PREFECT_TK}  
    env_file:
      - .env
The prefect token is saved in a .env file. The image can then be build with command:
Copy code
docker compose build
(sudo might be needed depending on how you installed docker)
🙌 1
c

Chris Reuter

07/24/2023, 11:26 AM
thank you for sharing @morten!
m

morten

07/24/2023, 11:27 AM
no problem 🙂 hopefully others can use it.
o

Oliver Mannion

07/25/2023, 7:16 AM
FYI - this will bake your API key into the image… To avoid this I pass `PREFECT_API_KEY`env var via docker compose.
m

morten

07/25/2023, 8:19 AM
thanks @Oliver Mannion do you mean something like this?
Copy code
services:
  worker:
    build:
      context: .
      args:
        PREFECT_TK: ${PREFECT_TK}  
    environment:
      PREFECT_TK
o

Oliver Mannion

07/25/2023, 8:33 AM
like this:
Copy code
services:
  worker:
    build:
      context: .
    environment:
      - PREFECT_API_KEY
      - PREFECT_API_URL
this will mean you need to have the following env vars set on the host, before running docker compose:
Copy code
export PREFECT_API_KEY='xxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
export PREFECT_API_URL='<https://api.prefect.cloud/api/accounts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/workspaces/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx>'
🙌 1
and you no longer need to do
prefect cloud login
in the Dockerfile
m

morten

07/25/2023, 8:37 AM
makes sense - much better!
right now I have set the
PREFECT_API_URL
to local host (if I remember correctly), but that environment variable with the link would automatically take if that
👍 1