Hello all, I'm running a Prefect flow in a Kubernetes cluster via DaskTaskRunner. My task is errori...

James Brady

11/08/2022, 5:19 PM

Hello all, I'm running a Prefect flow in a Kubernetes cluster via DaskTaskRunner. My task is erroring out with

Crash detected! Execution was interrupted by an unexpected exception.

(full logs in thread). I'm trying to figure out the best way to debug this further. In general – and to help diagnose this particular case – I'd like to shuttle logs to our DataDog instance: I found this discussion on the topic, but is there something more recent / more complete for Prefect v2? Aside from getting logs out, any other ideas for diagnosing this terse error?

James Brady

11/08/2022, 5:21 PM

Copy code

downloading file, entity_type=abstracts, download_url=<long_url>
05:38:13 PM
download_reshape_upload-8c37f7f7-48
reading JSON lines into DataFrame, local_file=/tmp/tmpxavexr8v
05:39:36 PM
download_reshape_upload-8c37f7f7-48
Crash detected! Execution was interrupted by an unexpected exception.

Mason Menges

11/09/2022, 5:48 PM

Hey James we have a Logging FAQ in discourse that covers a variety of different topics that may help in configuring logging for this https://discourse.prefect.io/t/prefect-logging-faq/1476

James Brady

11/10/2022, 3:56 PM

Thank you @Mason Menges! Checking that now

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.