Justin Trautmann

04/26/2023, 3:14 PM
hello community, hello prefect team, we've been using mlflow in one of out prefect flows and on prefect <=2.9.0 this worked fine. However after updating to prefect 2.10.x our deployments get stuck during import of the flow (in
). As far as I could debug this, this is due to the fact that mlflow messes with the python logger ( when being imported, removes all the handler and sets its own handlers. I am not sure why this problem didn't surface already with prefect <=2.9.0 but now with the reworked logging in 2.10 this is actually causing a deadlock. It seems like during the removal of the handler, the
is drained (by putting
in the
, which should be picked up by the
and cause the
to be set. But somewhere along this path, it gets stuck. I am still trying to come up with a minimal reproducible example but would be very interested in some general best practices around dependencies that manipulate the python logger. I assume that this could happen with a lot more dependencies than just mlfow and I feel like prefect should be able to wrap any logger modifications by flow dependencies to ensure that logs are visible in the cloud. Thanks