https://prefect.io logo
Join Slack
Powered by
hi I'm trying to run prefect2.0 in docker containe...
# prefect-community
m
hi I'm trying to run prefect2.0 in docker containers that I build manually when I run flows in a container, they seem to work I mount orion.db from host in docker run
Copy code
-v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
when I run orion dashboards from host:
Copy code
prefect orion start
it works as intended - shows flows run from container. But when I try to host dashboards from container I get empty dashboards (also mounting oriond.db, so both containers see the same database
Copy code
-v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
added -p 4200:4200 to map port fdrom host to container, and exported PREFECT_ORION_API_HOST=0.0.0.0 so I can access dashboards in container from my host. orion.db is properly mounted in both containers, because I see that it changes in them when i run a flow. Dashboards are accessible, but empty...
✅ 1
ok, probably my fault, incorrectly installed sqlite3 in container with dashbaords, checking... nah, installing sqlite3 in container that hosts dashboards didn't help
k
I tried with this and it works:
Copy code
docker run -it --rm \
    -p 4200:4200 \
    prefecthq/prefect:2.0b5-python3.8 \
        prefect orion start --host=0.0.0.0
That’s the same as yours?
m
i didn't use --host, just exported env to the container:
Copy code
PREFECT_ORION_API_HOST=0.0.0.0
and I used my image, not the official one, I will look into this And how do you deal with the database? Local prefect uses sqlite file (orion.db, by default from location ~/.prefect/orion.db) so I mount the file from host into containers to the path ~/.prefect/orion.db, so it gets persisted on the host, and all containers use the same file I try to make a setup with two containers, one hosting UI (prefect orion start) and one running flows (just python3 myflow.py) The container running flow works all right - writes into that file. UI run from the host (not from the container_ works fine - shows flows history. UI run from the container works, but shows no data
using --host option changes nothing, i think it's teh same as env PREFECT_ORION_API_HOST 😕
ok, it worked from the official image, will investigate this
so this works: (official image)
Copy code
docker run -it --rm     -p 4200:4200   -v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db  prefecthq/prefect:2.0b5-python3.8 prefect orion start --host=0.0.0.0
when I use my image it doesn't my image is basically python image with installed prefect and sqlite3: dockerfile is like this:
Copy code
FROM python:3.9.13-slim-buster

RUN mkdir /prefect/
ADD ./requirements.txt /prefect/
RUN apt-get update
RUN apt install sqlite3
run pip install -r /prefect/requirements.txt
ENTRYPOINT prefect orion start --host 0.0.0.0
and ./requirements.txt is
Copy code
prefect==2.0b5
prefect-snowflake==0.1.0
so something is missing here apparently
It's not a big problem, for I can use official image as in your example, Thank you for your help!! but I'm curious what is missing in "my image"
a
Marcin, why do you need to run this in Docker? If you want to run your flows in Docker, all you need is DockerFlowRunner, Orion doesn't have to run in Docker - in fact, NOT running Orion itself in Docker makes things way easier
you are likely stumbling upon network issues that are challenging to deal with in this setup
if you REALLY need that, it would be best to first create a docker network and run all those containers in the same Docker network, but I would strongly recommend running Orion in a virtual environment instead because everything gets soooo much easier that way (no mounts, network issues needed) and your flow runs can still be fully isolated in docker containers thanks to the DockerFlowRunner
running Orion in Docker adds a lot of negative engineering P
m
yeah, I'm just playing with prefect out of curiosity - I know it's not a preferred solution ;-) But as I said, running dashboard from your image works (when I mount orion.db properly) and running it from python docker image, that installs prefect using pip install, doesn't It got me confused and I just try to understand what's happening
a
sure, feel free to write some markdown Gist about your experience and share with the community - but for production, I would recommend running Orion without Docker since it makes everything much easier to maintain and troubleshoot already your issue here is a proof that running the orchestration layer itself in a Docker container adds a lot of overhead and wasted time which would be saved when running e.g. in a virtual environment + using DockerFlowRunner to run everything in a containerized fashion, you may move to Kubernetes
m
k8s will be the next step if we continue playing with prefect 👍
👍 1
4 Views
Open in Slack
PreviousNext
Powered by