Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
j
Jean-Marc LE FEVRE
02/18/2021, 3:05 PMWe are trying to use prefect with gevent, and we are having trouble with it. It seems that gevent monkey patch is triggered too late in the flow run. Does anyone have advice for this case ?
m
Michael Adkins
02/18/2021, 3:56 PMHi @Jean-Marc LE FEVRE -- could you give an example of what you're trying to do and the error?
j
Jean-Marc LE FEVRE
02/18/2021, 4:10 PMWell, we are trying to execute a task that have dependenciez with gevent
And we got this error! MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['pymongo.ssl_support (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/pymongo/ssl_support.py)', 'urllib3.util (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/urllib3/util/ssl_.py)'].
monkey.patch_all(aggressive=aggressive)
/home/jmlefevre/.virtualenvs/datavore3/lib/python3.7/site-packages/prefect/utilities/logging.py:129: UserWarning: Failed to write logs with error: RecursionError('maximum recursion depth exceeded while calling a Python object')
warnings.warn(f"Failed to write logs with error: {exc!r}")
This happens when a monkey patch after urllib import
m
Michael Adkins
02/18/2021, 6:02 PMAh I see. I'm not sure we're going to be able to work around this,
requestsOne idea: You could put the code that you want to use gevent in a python file and execute it with a
ShellTaskr
Rob Fowler
02/18/2021, 11:34 PMI ran into this issue. I switched to using dask itself to manage the asynchronous processes. Never looked back.
m
Michael Adkins
02/18/2021, 11:35 PMSweet Rob, do you have a simple example? I'm curious what the use-case is for this.
j
Jean-Marc LE FEVRE
03/01/2021, 1:54 PMHi guy
After a tons of tests, we are still not able to run prefect with gevent
Even shell mode need to have monkey_patch() before importing urllib
When we modify the agent to use monkey patch before importing urllib, it work ... But this is not suitable for production imho