Hello all, when using the dask executor is there any way to control what prefect writes to the console log? We are getting a line in there per prefect log message sent. We send a lot of log messages and it is causing performance problems. Is there any way to turn that off so that the worker doesn’t have to spend the overhead of interleaving lines?

You can filter logs with certain messages like this and the Prefect logger won’t emit them

awesome thanks

I think that is hard. I would try to add the filter on the worker side lol. I really don’t immediately know because you could fiddle with the log level, but I don’t think that’s what you want either

I guess it depends on where the logs are being sent to prefect server. If they are being sent by the thread/process that generates them, then having them all sent to the worker so that they can be printed (and then ignored) is very wasteful. But if they are all being sent from a central location then filtering them out isn’t a big deal.

I think for DaskExecutor specifically worker logs are not sent to Prefect right?

I’m talking about the logs generated by tasks using prefect.contextd.get(“logger”). For ever line that is in one of the logs I see on prefect web UI, there is also a line in the console output of dask worker.

I think I would personally try removing the handlers inside the task. For the Prefect one, you can try removing the CloudHandler. I don’t know though