Is there a way to monitor for lack of logs in a pr...
# prefect-community
k
Is there a way to monitor for lack of logs in a prefect task (or better, in a flow)?
k
what do you mean by the lack of logs?
k
A process gets stuck and stops output either to stdout or to a file
Essentially, a way to add a watchdog over a log file/stream in prefect
👍 1
Any ideas @Khuyen Tran?
k
k
It's not about capturing the data. Sometimes our process freezes, and we want to restart the task if that happens. Instead of creating a hand-crafted solution, it'd be nice to reuse some solution that prefect could provide
k
could you be more specific about what you mean by the process freezes?
k
We use a prefect task to start a process. The process outputs some log on stdout every once a while to show that it's alive. Due to some quirk of our setup, once in 100s of runs, the process sometimes just ... stops progressing, and ends up stuck. No output on stdout, no cpu usage, no signal to terminate/kill. Since the sole detection is via lack of stdout, if we can set a wait on stdout (eg: if no stdout in 50 seconds, restart task), that'd be helpful. GitHub and GitLab both have this feature
k
It seems like you can set task-level timeout to achieve what you describe
k
That wouldn't work. Imagine a task that normally runs for 2 days and gets stuck after just 2 hours. We can't set a 2 day timeout. That'd be nearly useless
Can I assume this is impossible in prefect?