Hey I m finding `logger` doesn t seem to output logs either

Question

Hey I m finding `logger` doesn t seem to output logs either to terminal or to the cloud UI when using the `DaskTaskRunner ` I m calling `get run logger ` within the task code attached in thread Any ideas am I missing something works fine with sequential concurrent task runners just not the Dask one

David Elliott · Answer

Untitled py

David Elliott · Answer

run local output sh

David Elliott · Answer

Prefect version 2 4 2 prefect dask==0 2 0

Zanie · Answer

This is a known issue unfortunately

David Elliott · Answer

Ah ok Do you think it ll be worked on for a future version and does it affect Dask on kubernetes as well as locally

Zanie · Answer

Sorry got distracted in the middle of my follow up message smile

Zanie · Answer

Basically Logging is configured per process this is just how Python does it and we need to ensure that your logging configuration has been applied to the given process at the start of each run However you can t configure logging more than once or weird things happen So we re in this weird place where we store a global variable indicating if logging has already been configured and just ignore future calls to apply the logging config if its set

Zanie · Answer

I suspect that this global variable is getting passed to Dask workers somehow so logging is not reconfigured on a new process

Zanie · Answer

I m not really sure though it s on the end of a long todo list to investigate

Zanie · Answer

It ll definitely work on a future version I think this affects any Dask setup

David Elliott · Answer

Cool thanks for explaining It might be a bit of a blocker for us migrating 1 0 gt 2 0 if we can t log our task logs to cloud so hoping you are able to get to the bottom of it at some stage we re quite near the upper limit of 3000 tasks in 1 0 so were hoping to migrate in the coming weeks grimacing

Zanie · Answer

What do you need Dask for

David Elliott · Answer

So we can execute a lot of SQL scripts in parallel not concurrently atm with 1 0 we have Dask on k8s 4x worker pods 16 threads each running up to 64 SQL scripts in parallel Just noticed the Ray task runner which I ve not heard of any idea if there are similar logger issues there and whether it works on k8s Not wedded to dask if there s an alternative that can allow us to run tasks in parallel on k8s

Robert Hales · Answer

Hi there may have some info that will be helpful when looking into this Dask does not want to log when the worker is running locally however if you move it to a different machine even a docker container logs begin to work

David Elliott · Answer

Ohh interesting I m actually just about to start testing on k8s running within a docker container so will see how it looks there thanks

Robert Hales · Answer

I was happily surprised when my logs where coming through when running in ecs though it was going to block me