https://prefect.io logo
Title
j

Justin Trautmann

04/26/2023, 3:14 PM
hello community, hello prefect team, we've been using mlflow in one of out prefect flows and on prefect <=2.9.0 this worked fine. However after updating to prefect 2.10.x our deployments get stuck during import of the flow (in
load_flow_from_entrypoint
). As far as I could debug this, this is due to the fact that mlflow messes with the python logger (https://github.com/mlflow/mlflow/issues/4957) when being imported, removes all the handler and sets its own handlers. I am not sure why this problem didn't surface already with prefect <=2.9.0 but now with the reworked logging in 2.10 this is actually causing a deadlock. It seems like during the removal of the handler, the
QueueService
is drained (by putting
None
in the
_queue
, which should be picked up by the
_main_loop
and cause the
_done_event
to be set. But somewhere along this path, it gets stuck. I am still trying to come up with a minimal reproducible example but would be very interested in some general best practices around dependencies that manipulate the python logger. I assume that this could happen with a lot more dependencies than just mlfow and I feel like prefect should be able to wrap any logger modifications by flow dependencies to ensure that logs are visible in the cloud. Thanks