https://prefect.io logo
Title
k

Kunal Tyagi

10/05/2022, 3:21 PM
Is there a way to monitor for lack of logs in a prefect task (or better, in a flow)?
k

Khuyen Tran

10/05/2022, 3:21 PM
what do you mean by the lack of logs?
k

Kunal Tyagi

10/05/2022, 3:22 PM
A process gets stuck and stops output either to stdout or to a file
Essentially, a way to add a watchdog over a log file/stream in prefect
👍 1
Any ideas @Khuyen Tran?
k

Khuyen Tran

10/07/2022, 2:18 PM
k

Kunal Tyagi

10/08/2022, 1:03 AM
It's not about capturing the data. Sometimes our process freezes, and we want to restart the task if that happens. Instead of creating a hand-crafted solution, it'd be nice to reuse some solution that prefect could provide
k

Khuyen Tran

10/10/2022, 1:13 PM
could you be more specific about what you mean by the process freezes?
k

Kunal Tyagi

10/10/2022, 1:25 PM
We use a prefect task to start a process. The process outputs some log on stdout every once a while to show that it's alive. Due to some quirk of our setup, once in 100s of runs, the process sometimes just ... stops progressing, and ends up stuck. No output on stdout, no cpu usage, no signal to terminate/kill. Since the sole detection is via lack of stdout, if we can set a wait on stdout (eg: if no stdout in 50 seconds, restart task), that'd be helpful. GitHub and GitLab both have this feature
k

Khuyen Tran

10/10/2022, 7:14 PM
It seems like you can set task-level timeout to achieve what you describe
k

Kunal Tyagi

10/11/2022, 2:53 AM
That wouldn't work. Imagine a task that normally runs for 2 days and gets stuck after just 2 hours. We can't set a 2 day timeout. That'd be nearly useless
Can I assume this is impossible in prefect?