Hello Is it possible to stream task runner initialisation lo

Question

Hello Is it possible to stream task runner initialisation logs to the Cloud UI logs I am using DaskTaskRunner and user can t see logs about cluster creation and dashboard url

Vadym Dytyniak · Answer

And actually errors are very abstract I can see more details only in pod logs

Christopher Boyd · Answer

Hi Vadym do you have task loggers in your prefect tasks DaskTaskRunners will can collect logs into the Prefect UI although it is slightly delayed just because of the nature of the processes having to spin up tear down log ship and batch write to the cloud

Christopher Boyd · Answer

but if you are logging inside the task with a task logger then you should see logs in the UI although they can be delayed

Vadym Dytyniak · Answer

There is no logs from DaskTaskRunner even after task is completed

Christopher Boyd · Answer

Do you have a logger inside the task

Vadym Dytyniak · Answer

I am not sure I understand your question DaskTaskRunner starts before all tasks

Christopher Boyd · Answer

Christopher Boyd · Answer

You can have a task logger and a flow logger task name= log example task def logger task logger = get run logger <http logger info|logger info> INFO level log message from a task flow name= log example flow def logger flow logger task

Christopher Boyd · Answer

If you don t have a task logger inside the task for Dask logs are not being surfaced for that task

Vadym Dytyniak · Answer

I understand it I am asking about <https github com PrefectHQ prefect dask blob d651a3c6d30421c4393396dcf087930040013c54 prefect dask task runners py L293>

Christopher Boyd · Answer

Gotcha I completely misundertood

Christopher Boyd · Answer

what does your task definition look like

Christopher Boyd · Answer

It looks like that should be adding info logs to the logger

Christopher Boyd · Answer

before the connection is made

Vadym Dytyniak · Answer

``` task def write data storage Storage effective date datetime date gt None df = pd DataFrame col 1 1 2 3 col 2 1 2 3 df date = effective date storage save dd from pandas df npartitions=1 SAMPLE NAMESPACE ``` in the task I don t have access to the cluster I just submit it to the task runner I specified in flow attributes

Christopher Boyd · Answer

where is the dask cluster spun up from the task runner

Vadym Dytyniak · Answer

Does it matter I can pass any cluster class

Vadym Dytyniak · Answer

```DaskTaskRunner cluster class= dask cloudprovider aws FargateCluster cluster kwargs= n workers 4 image my prefect image ```

Vadym Dytyniak · Answer

or pass existing address ``` flow task runner=DaskTaskRunner address= <http my dask cluster> def my flow ```

Vadym Dytyniak · Answer

The main question how to see logs from DaskTaskRunner in Cloud UI logs

Vadym Dytyniak · Answer

Because I see them on prod pod I am running the flow

Christopher Boyd · Answer

That s what I m trying to suggest when you call the task runner you need to log the output of where it s being called from

Christopher Boyd · Answer

the taskrunner itself logs but if you don t have a logger defined in your flow where it is being instantiated

Christopher Boyd · Answer

there is nothing surfacing it

Christopher Boyd · Answer

```Logging in flows To log from a flow retrieve a logger instance with get run logger then call the standard Python logging methods from prefect import flow get run logger flow name= log example flow def logger flow logger = get run logger <http logger info|logger info> INFO level log message Prefect automatically uses the flow run logger based on the flow context If you run the above code Prefect captures the following as a log event 15 35 17 304 | INFO | Flow run mottled marten INFO level log message The default flow run log formatter uses the flow run name for log messages ```

Christopher Boyd · Answer

the DaskTaskRunner is surfacing info logs so if you are creating that in your flow then I believe you would need a flow logger context to see those logs

Vadym Dytyniak · Answer

but as I understand task runner starts before any code in the flow function

Vadym Dytyniak · Answer

I see these logs in pod logs but not in Cloud UI