Avi A06/30/2020, 8:22 PM
in my flow and getting errors from GCS (caused by the
object trying to store task results:
The reason is (probably) that I have many mapped tasks, and they are running in parallel. It’s a 32-core machine, but I can’t really figure out how Prefect/Dask decides how many run in parallel. Sometimes it’s more than 32. The bigger problem here is that I think it caused the flow to get stuck, so I didn’t even have the indication for the failure and couldn’t go and restart it. Anyhow, any suggestions on how to overcome this? Or at the very least cause my flow to fail on such things so that I can restart? Does it have anything to do with the fact that these tasks have a retry option?
Unexpected error: SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://storage.googleapis.com|storage.googleapis.com>', port=443): Max retries exceeded with url: /upload/storage/v1/b/prefect-bucket/o?uploadType=multipart (Caused by SSLError(OSError(24, 'Too many open files')))"))
Jim Crist-Harif06/30/2020, 8:24 PM
. If you don't see an obvious issue in your code, would you mind creating an issue https://github.com/PrefectHQ/prefect/issues? Prefect should be able to handle this use case just fine, so if it's not it's a bug.
Avi A06/30/2020, 8:38 PM
which is a pluging for pyfilesystem2 that allows me to open GCS and local files interchangeably. I have things like this in the code:
return fs.open_fs(dir_path, create=False).exists(SUCCESS)
is a GCS path, so I’m probably not closing it correctly. But to my defense, to the best of my knowledge `cpython`’s garbage collector should close these cursors automatically (as opposed to
implementation). I’ll try and report back
Jim Crist-Harif06/30/2020, 10:26 PM
Avi A06/30/2020, 10:27 PM
architecture. I guess it’s time to decommission it but I need to find time for that 😕