https://prefect.io logo
Join SlackCommunities
Powered by
Hi! I am trying to run `prefect==2.0a5` in Docker....
# ask-community
i
Hi! I am trying to run 
prefect==2.0a5
in Docker. Here is my Dockerfile:
Copy code
FROM python:3.8
RUN adduser prefect
USER prefect
WORKDIR /home/prefect
COPY --chown=prefect:prefect <http://requirements.in|requirements.in> ./
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PATH="/home/prefect/.local/bin:${PATH}"
RUN pip install --user --no-cache-dir -r <http://requirements.in|requirements.in>
COPY --chown=prefect:prefect flow.py flow_deployment.py ./
Inside the container I create a deployment and several flow runs, but all of them are marked as “late”. Here is the terminal output:
Copy code
$ prefect orion start --host 0.0.0.0 --log-level DEBUG
Starting Orion API server...
INFO:     Started server process [71]
INFO:     Waiting for application startup.
09:54:06.189 | Scheduler service scheduled to start in-app
09:54:06.189 | MarkLateRuns service scheduled to start in-app
INFO:     Application startup complete.
INFO:     Uvicorn running on <http://0.0.0.0:4200> (Press CTRL+C to quit)
09:54:06.501 | Finished monitoring for late runs.
09:54:06.538 | Scheduled 0 runs.
Starting agent connected to <http://0.0.0.0:4200/api/>...
Agent started! Checking for flow runs...
09:54:07.298 | Submitting flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa'
09:54:07.298 | Submitting flow run '521624eb-cec8-4f9b-9e92-f203e104586a'
09:54:07.298 | Submitting flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf'
09:54:07.299 | Submitting flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51'
09:54:07.299 | Submitting flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9'
09:54:07.300 | Completed submission of flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa'
09:54:07.300 | Completed submission of flow run '521624eb-cec8-4f9b-9e92-f203e104586a'
09:54:07.300 | Completed submission of flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf'
09:54:07.300 | Completed submission of flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51'
09:54:07.300 | Completed submission of flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9'
09:54:08.969 | Flow run '521624eb-cec8-4f9b-9e92-f203e104586a' exited with exception: KeyError('__main__')
09:54:08.975 | Flow run '2895bdfa-082c-43b5-afc2-d0dcc269bf51' exited with exception: KeyError('__main__')
09:54:08.976 | Flow run 'f25da7b6-7893-4107-aaa3-df22377e2ccf' exited with exception: KeyError('__main__')
09:54:08.979 | Flow run '2b445dbd-58b4-4acb-89b7-1f6782dc0ec9' exited with exception: KeyError('__main__')
09:54:08.980 | Flow run 'f0855bd3-2eab-4346-ad3a-2e237a688faa' exited with exception: KeyError('__main__')
Is there a way to debug these KeyErrors?
a
@Ivan Zaikin so far, Orion only provides local deployments and we haven’t yet implemented dockerized deployments. The KeyError indicates you are using DaskExecutor within that Docker container, is it true? If so, disabling multiprocessing may fix it:
Copy code
DaskExecutor(cluster_kwargs=dict(processes=False))
i
I haven’t specified DaskExecutor anywhere. How can I check what executor is being used?
a
I see, then ignore the executor suggestion. You can try to execute the deployment manually:
Copy code
prefect deployment execute deployment_name
Does this trigger a deployment without issues?
i
Copy code
/home/prefect/.local/lib/python3.8/site-packages/prefect/cli/deployment.py:115: RuntimeWarning: coroutine 'create_then_begin_flow_run' was never awaited
  flow(**parameters)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
I am using an async flow.
a
Ok. I would try all the different flow patterns incl. async in a local process first 🙂 running it all in Docker makes it more difficult to debug, since so far Orion has been tested and released with local deployments only. Feel free to share your flow that causes the error and I could check if I can reproduce the error in my local environment.
i
Here is a simplified version of my flow that causes the same error locally:
Copy code
» prefect deployment execute 'jira/daily'
/Users/i/.pyenv/versions/3.8.10/envs/prefect/lib/python3.8/site-packages/prefect/cli/deployment.py:115: RuntimeWarning: coroutine 'create_then_begin_flow_run' was never awaited
  flow(**parameters)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
test_flow.pytest_flow_deployment.py
But instead of the KeyError I am getting this error locally when running `prefect orion start`:
Untitled.txt
@Anna Geller ⬆️ here are my flow and an error I got locally.
a
Thanks for sharing it! Can you tell which sqlite3 version do you use?
I will consult with the team since I don’t understand why you get this error.
i
I am on Mac OS
Copy code
» sqlite3 --version
3.28.0 2019-04-15 14:49:49 378230ae7f4b721c8b8d83c8ceb891449685cd23b1702a57841f1be40b5daapl
a
thanks a lot! will get back to you once I know more
🙏 1
The only advice I would have would be to try resetting the DB since it often helps with various DB issues on Orion end. If that’s possible, you could do:
Copy code
prefect orion reset-db
and then try to rerun the flow and restart Orion. I will nevertheless get back to you once I got response from the team.
i
I already did this
👍 1
a
@Ivan Zaikin Looking at this again, I think this might be just an issue with a sqlite3 version. Your version is from 2019. My version that got automatically installed with Orion was:
Copy code
3.36.0 2021-06-18
Can you try to upgrade sqlite, reset the db, restart Orion and see if that was the issue?
👀 1
Do you run it in a virtual environment? Perhaps you could create a new virtual environment and do a clean fresh install of both Orion and sqlite again.
i
Ok, after upgrading sqlite3 I started getting the same KeyError as in Docker:
Copy code
22:25:21.706 | Submitting flow run '1425a95e-059a-4cb9-9898-082539e04e3a'
22:25:21.706 | Completed submission of flow run '1425a95e-059a-4cb9-9898-082539e04e3a'
22:25:22.742 | Flow run '1425a95e-059a-4cb9-9898-082539e04e3a' exited with exception: KeyError('__main__')
22:25:26.096 | Finished monitoring for late runs.
z
Hi! I want to dig into this but I’m pretty busy so it may be slow
What’s 
pip show distributed
give?
i
@Zanie:
Copy code
» pip show distributed                                                                    
Name: distributed
Version: 2021.11.2
Summary: Distributed scheduler for Dask
Home-page: <https://distributed.dask.org>
Author: 
Author-email: 
License: BSD
Location: /Users/i/.pyenv/versions/3.10.0/envs/prefect-py310/lib/python3.10/site-packages
Requires: click, cloudpickle, dask, jinja2, msgpack, psutil, pyyaml, setuptools, sortedcontainers, tblib, toolz, tornado, zict
Required-by: prefect
@Zanie, any news on this?
@Anna Geller I am getting exactly the same errors on 
prefect==2.0a6
z
Hey Ivan, I’ve reproduced this myself and will start investigating
I believe this will be fixed in the next release. This was in issue with anyio worker processes and dask distributed compatibility
🙏 2
i
I did not mean to use dask, honestly. I have no idea why I got a dask-related error.
z
Oh interesting, looking at your test flow I see you’re not using the dask executor / task runner.
Perhaps it was not dask specific after all, but it should be fixed in the next version anyway.
6 Views
Open in Slack
PreviousNext
Powered by