https://prefect.io
Join Slack
We are trying to use prefect with gevent, and we are having trouble with it. It seems that gevent mo...
# ask-community
j
We are trying to use prefect with gevent, and we are having trouble with it. It seems that gevent monkey patch is triggered too late in the flow run. Does anyone have advice for this case ?
z
Hi @Jean-Marc LE FEVRE -- could you give an example of what you're trying to do and the error?
j
Well, we are trying to execute a task that have dependenciez with gevent
And we got this error! MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['pymongo.ssl_support (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/pymongo/ssl_support.py)', 'urllib3.util (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/ssl_.py)']. monkey.patch_all(aggressive=aggressive) /home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/prefect/utilities/logging.py129 UserWarning: Failed to write logs with error: RecursionError('maximum recursion depth exceeded while calling a Python object') warnings.warn(f"Failed to write logs with error: {exc!r}")
This happens when a monkey patch after urllib import
z
Ah I see. I'm not sure we're going to be able to work around this, 
requests
is imported by the flow runner for communication with the API and monkeypatching a library that we're using seems risky. I'll talk to the team and see if anyone has any ideas.
One idea: You could put the code that you want to use gevent in a python file and execute it with a 
ShellTask
instead -- that way it's isolated from prefect's runtime.
r
I ran into this issue. I switched to using dask itself to manage the asynchronous processes. Never looked back.
z
Sweet Rob, do you have a simple example? I'm curious what the use-case is for this.
j
Hi guy
After a tons of tests, we are still not able to run prefect with gevent
Even shell mode need to have monkey_patch() before importing urllib
When we modify the agent to use monkey patch before importing urllib, it work ... But this is not suitable for production imho
276 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by