https://prefect.io logo
#prefect-community
Title
# prefect-community
d

Daniel Komisar

02/25/2022, 9:14 PM
Hello all, when using the dask executor is there any way to control what prefect writes to the console log? We are getting a line in there per prefect log message sent. We send a lot of log messages and it is causing performance problems. Is there any way to turn that off so that the worker doesn’t have to spend the overhead of interleaving lines?
k

Kevin Kho

02/25/2022, 9:16 PM
You can filter logs with certain messages like this and the Prefect logger won’t emit them
d

Daniel Komisar

02/25/2022, 9:16 PM
awesome thanks
Sorry my mistake this isn’t what I was asking. I want the logs to be emitted to prefect. I don’t want them to be printed by the dask worker to it’s stdout.
k

Kevin Kho

02/25/2022, 9:22 PM
I think that is hard. I would try to add the filter on the worker side lol. I really don’t immediately know because you could fiddle with the log level, but I don’t think that’s what you want either
d

Daniel Komisar

02/25/2022, 9:25 PM
I guess it depends on where the logs are being sent to prefect server. If they are being sent by the thread/process that generates them, then having them all sent to the worker so that they can be printed (and then ignored) is very wasteful. But if they are all being sent from a central location then filtering them out isn’t a big deal.
k

Kevin Kho

02/25/2022, 9:28 PM
I think for DaskExecutor specifically worker logs are not sent to Prefect right?
d

Daniel Komisar

02/25/2022, 9:30 PM
I’m talking about the logs generated by tasks using prefect.contextd.get(“logger”). For ever line that is in one of the logs I see on prefect web UI, there is also a line in the console output of dask worker.
There is other stuff in there I’d like to collect, but with the log lines it’s too much. I was also hoping to turn it off instead of filtering so it wouldn’t be doing as much work but maybe that’s not possible.
k

Kevin Kho

02/25/2022, 9:32 PM
I think I would personally try removing the handlers inside the task. For the Prefect one, you can try removing the CloudHandler. I don’t know though