We are trying to use prefect with gevent, and we a...
# prefect-communityj
Jean-Marc LE FEVRE
02/18/2021, 3:05 PMWe are trying to use prefect with gevent, and we are having trouble with it. It seems that gevent monkey patch is triggered too late in the flow run. Does anyone have advice for this case ?
z
Zanie
02/18/2021, 3:56 PM
Hi @Jean-Marc LE FEVRE -- could you give an example of what you're trying to do and the error?
j
Jean-Marc LE FEVRE
02/18/2021, 4:10 PMWell, we are trying to execute a task that have dependenciez with gevent
Jean-Marc LE FEVRE
02/18/2021, 4:10 PMAnd we got this error! MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['pymongo.ssl_support (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/pymongo/ssl_support.py)', 'urllib3.util (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/ssl_.py)'].
monkey.patch_all(aggressive=aggressive)
/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/prefect/utilities/logging.py129 UserWarning: Failed to write logs with error: RecursionError('maximum recursion depth exceeded while calling a Python object')
warnings.warn(f"Failed to write logs with error: {exc!r}")
Jean-Marc LE FEVRE
02/18/2021, 4:11 PMThis happens when a monkey patch after urllib import
z
Zanie
02/18/2021, 6:02 PM
Ah I see. I'm not sure we're going to be able to work around this,
requestsZanie
02/18/2021, 6:04 PM
One idea: You could put the code that you want to use gevent in a python file and execute it with a
ShellTaskr
Rob Fowler
02/18/2021, 11:34 PMI ran into this issue. I switched to using dask itself to manage the asynchronous processes. Never looked back.
z
Zanie
02/18/2021, 11:35 PM
Sweet Rob, do you have a simple example? I'm curious what the use-case is for this.
j
Jean-Marc LE FEVRE
03/01/2021, 1:54 PMHi guy
Jean-Marc LE FEVRE
03/01/2021, 1:55 PMAfter a tons of tests, we are still not able to run prefect with gevent
Jean-Marc LE FEVRE
03/01/2021, 1:55 PMEven shell mode need to have monkey_patch() before importing urllib
Jean-Marc LE FEVRE
03/01/2021, 1:56 PMWhen we modify the agent to use monkey patch before importing urllib, it work ... But this is not suitable for production imho
197 Views