hi I m trying to run prefect2 0 in docker containers that I

Question

hi I m trying to run prefect2 0 in docker containers that I build manually when I run flows in a container they seem to work I mount orion db from host in docker run ``` v home marcin prefect orion db root prefect orion db``` when I run orion dashboards from host ```prefect orion start``` it works as intended shows flows run from container But when I try to host dashboards from container I get empty dashboards also mounting oriond db so both containers see the same database ``` v home marcin prefect orion db root prefect orion db``` added p 4200 4200 to map port fdrom host to container and exported PREFECT ORION API HOST=0 0 0 0 so I can access dashboards in container from my host orion db is properly mounted in both containers because I see that it changes in them when i run a flow Dashboards are accessible but empty

Marcin Grzybowski · Answer

~ok probably my fault incorrectly installed sqlite3 in container with dashbaords checking ~ nah installing sqlite3 in container that hosts dashboards didn t help

Marcin Grzybowski · Answer

seems similar to this <https prefect community slack com archives C0192RWGJQH p1651758271890979>

Kevin Kho · Answer

I tried with this and it works ```docker run it rm p 4200 4200 prefecthq prefect 2 0b5 python3 8 prefect orion start host=0 0 0 0``` That s the same as yours

Marcin Grzybowski · Answer

i didn t use host just exported env to the container ```PREFECT ORION API HOST=0 0 0 0``` and I used my image not the official one I will look into this And how do you deal with the database Local prefect uses sqlite file orion db by default from location ~ prefect orion db so I mount the file from host into containers to the path ~ prefect orion db so it gets persisted on the host and all containers use the same file I try to make a setup with two containers one hosting UI prefect orion start and one running flows just python3 myflow py The container running flow works all right writes into that file UI run from the host not from the container works fine shows flows history UI run from the container works but shows no data

Marcin Grzybowski · Answer

using host option changes nothing i think it s teh same as env PREFECT ORION API HOST confused

Marcin Grzybowski · Answer

ok it worked from the official image will investigate this

Marcin Grzybowski · Answer

so this works official image ``` docker run it rm p 4200 4200 v home marcin prefect orion db root prefect orion db prefecthq prefect 2 0b5 python3 8 prefect orion start host=0 0 0 0``` when I use my image it doesn t my image is basically python image with installed prefect and sqlite3 dockerfile is like this ```FROM python 3 9 13 slim buster RUN mkdir prefect ADD requirements txt prefect RUN apt get update RUN apt install sqlite3 run pip install r prefect requirements txt ENTRYPOINT prefect orion start host 0 0 0 0``` and requirements txt is ```prefect==2 0b5 prefect snowflake==0 1 0``` so something is missing here apparently

Marcin Grzybowski · Answer

It s not a big problem for I can use official image as in your example Thank you for your help but I m curious what is missing in my image

Anna Geller · Answer

Marcin why do you need to run this in Docker If you want to run your flows in Docker all you need is DockerFlowRunner Orion doesn t have to run in Docker in fact NOT running Orion itself in Docker makes things way easier

Anna Geller · Answer

you are likely stumbling upon network issues that are challenging to deal with in this setup

Anna Geller · Answer

if you REALLY need that it would be best to first create a docker network and run all those containers in the same Docker network but I would strongly recommend running Orion in a virtual environment instead because everything gets soooo much easier that way no mounts network issues needed and your flow runs can still be fully isolated in docker containers thanks to the DockerFlowRunner

Anna Geller · Answer

running Orion in Docker adds a lot of negative engineering prefect

Marcin Grzybowski · Answer

yeah I m just playing with prefect out of curiosity I know it s not a preferred solution But as I said running dashboard from your image works when I mount orion db properly and running it from python docker image that installs prefect using pip install doesn t It got me confused and I just try to understand what s happening

Anna Geller · Answer

sure feel free to write some markdown Gist about your experience and share with the community but for production I would recommend running Orion without Docker since it makes everything much easier to maintain and troubleshoot already your issue here is a proof that running the orchestration layer itself in a Docker container adds a lot of overhead and wasted time which would be saved when running e g in a virtual environment + using DockerFlowRunner to run everything in a containerized fashion you may move to Kubernetes

Marcin Grzybowski · Answer

k8s will be the next step if we continue playing with prefect +1