m

    Marcin Grzybowski

    3 months ago
    hi I'm trying to run prefect2.0 in docker containers that I build manually when I run flows in a container, they seem to work I mount orion.db from host in docker run
    -v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
    when I run orion dashboards from host:
    prefect orion start
    it works as intended - shows flows run from container. But when I try to host dashboards from container I get empty dashboards (also mounting oriond.db, so both containers see the same database
    -v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
    added -p 4200:4200 to map port fdrom host to container, and exported PREFECT_ORION_API_HOST=0.0.0.0 so I can access dashboards in container from my host. orion.db is properly mounted in both containers, because I see that it changes in them when i run a flow. Dashboards are accessible, but empty...
    ok, probably my fault, incorrectly installed sqlite3 in container with dashbaords, checking... nah, installing sqlite3 in container that hosts dashboards didn't help
    Kevin Kho

    Kevin Kho

    3 months ago
    I tried with this and it works:
    docker run -it --rm \
        -p 4200:4200 \
        prefecthq/prefect:2.0b5-python3.8 \
            prefect orion start --host=0.0.0.0
    That’s the same as yours?
    m

    Marcin Grzybowski

    3 months ago
    i didn't use --host, just exported env to the container:
    PREFECT_ORION_API_HOST=0.0.0.0
    and I used my image, not the official one, I will look into this And how do you deal with the database? Local prefect uses sqlite file (orion.db, by default from location ~/.prefect/orion.db) so I mount the file from host into containers to the path ~/.prefect/orion.db, so it gets persisted on the host, and all containers use the same file I try to make a setup with two containers, one hosting UI (prefect orion start) and one running flows (just python3 myflow.py) The container running flow works all right - writes into that file. UI run from the host (not from the container_ works fine - shows flows history. UI run from the container works, but shows no data
    using --host option changes nothing, i think it's teh same as env PREFECT_ORION_API_HOST πŸ˜•
    ok, it worked from the official image, will investigate this
    so this works: (official image)
    docker run -it --rm     -p 4200:4200   -v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db  prefecthq/prefect:2.0b5-python3.8 prefect orion start --host=0.0.0.0
    when I use my image it doesn't my image is basically python image with installed prefect and sqlite3: dockerfile is like this:
    FROM python:3.9.13-slim-buster
    
    RUN mkdir /prefect/
    ADD ./requirements.txt /prefect/
    RUN apt-get update
    RUN apt install sqlite3
    run pip install -r /prefect/requirements.txt
    ENTRYPOINT prefect orion start --host 0.0.0.0
    and ./requirements.txt is
    prefect==2.0b5
    prefect-snowflake==0.1.0
    so something is missing here apparently
    It's not a big problem, for I can use official image as in your example, Thank you for your help!! but I'm curious what is missing in "my image"
    Anna Geller

    Anna Geller

    3 months ago
    Marcin, why do you need to run this in Docker? If you want to run your flows in Docker, all you need is DockerFlowRunner, Orion doesn't have to run in Docker - in fact, NOT running Orion itself in Docker makes things way easier
    you are likely stumbling upon network issues that are challenging to deal with in this setup
    if you REALLY need that, it would be best to first create a docker network and run all those containers in the same Docker network, but I would strongly recommend running Orion in a virtual environment instead because everything gets soooo much easier that way (no mounts, network issues needed) and your flow runs can still be fully isolated in docker containers thanks to the DockerFlowRunner
    running Orion in Docker adds a lot of negative engineering πŸ˜›refect:
    m

    Marcin Grzybowski

    3 months ago
    yeah, I'm just playing with prefect out of curiosity - I know it's not a preferred solution πŸ˜‰ But as I said, running dashboard from your image works (when I mount orion.db properly) and running it from python docker image, that installs prefect using pip install, doesn't It got me confused and I just try to understand what's happening
    Anna Geller

    Anna Geller

    3 months ago
    sure, feel free to write some markdown Gist about your experience and share with the community - but for production, I would recommend running Orion without Docker since it makes everything much easier to maintain and troubleshoot already your issue here is a proof that running the orchestration layer itself in a Docker container adds a lot of overhead and wasted time which would be saved when running e.g. in a virtual environment + using DockerFlowRunner to run everything in a containerized fashion, you may move to Kubernetes
    m

    Marcin Grzybowski

    3 months ago
    k8s will be the next step if we continue playing with prefect πŸ‘