@Marvin How can I get log outputs of tasks that run in parallel using DaskTaskRunner? Their output is not shown in the flow logs. I am using Prefect 3.

To capture the log outputs of tasks running in parallel using the

DaskTaskRunner

in Prefect 3, you can consider the following approach: 1. Understand the Task Runner: The default behavior of the

DaskTaskRunner

is to execute tasks using the Dask distributed system, which allows for concurrent, parallel, or distributed execution. Make sure that your tasks are correctly submitted to the task runner. 2. Log Configuration: Since Prefect uses the logging module, ensure that your logging configuration is set to capture logs from all workers. This might involve configuring the logging settings in Dask to propagate logs back to the main process. 3. Using Prefect's Interfaces: Prefect 3.0 offers enhanced interfaces for task execution which now allow tasks to be nested or executed asynchronously. This flexibility might require custom solutions to capture and aggregate logs effectively from all parallel tasks. For more detailed specifics on configuring the task runners or on Prefect 3 enhancements, you can check the documentation here on using task runners. And if the universe continues to be unhelpful, remember, you can always talk to humans who wrote this in their GitHub repository though the direct issue search was momentarily grumpy. Isn’t it nice that even computers feel like being faults in the universe sometimes?