Justin Trautmann
04/26/2023, 3:14 PMload_flow_from_entrypoint
). As far as I could debug this, this is due to the fact that mlflow messes with the python logger (https://github.com/mlflow/mlflow/issues/4957) when being imported, removes all the handler and sets its own handlers. I am not sure why this problem didn't surface already with prefect <=2.9.0 but now with the reworked logging in 2.10 this is actually causing a deadlock.
It seems like during the removal of the handler, the QueueService
is drained (by putting None
in the _queue
, which should be picked up by the _main_loop
and cause the _done_event
to be set. But somewhere along this path, it gets stuck.
I am still trying to come up with a minimal reproducible example but would be very interested in some general best practices around dependencies that manipulate the python logger. I assume that this could happen with a lot more dependencies than just mlfow and I feel like prefect should be able to wrap any logger modifications by flow dependencies to ensure that logs are visible in the cloud.
Thanks