Is there a way to get the logs from a single element of a mapped task?
k
Kevin Kho
05/22/2021, 10:06 PM
Hey @Jeremy Phelps, are you using a DaskExecutor?
j
Jeremy Phelps
05/22/2021, 10:07 PM
Yes.
Jeremy Phelps
05/22/2021, 10:09 PM
Looks like the error is being reported as a "state message" instead of being put into the log stream.
k
Kevin Kho
05/22/2021, 10:09 PM
In general no there isn’t a way because Dask itself does not move the logs around between client and workers so we don’t innately get worker logs. Even if the Prefect logger is configured to send stuff to the backend, the Python logger gets reinstantiated when it gets to Dask workers. The users who really need to get the logs from Dask workers set up their own service to collect it.
j
Jeremy Phelps
05/22/2021, 10:10 PM
Why is Prefect able to get any logs, then?
k
Kevin Kho
05/22/2021, 10:12 PM
You can see this. That’s from the client side logs.
Kevin Kho
05/22/2021, 10:12 PM
There is a thread with more info somewhere. Looking for it.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.