Hello, several times already I met a problem when ...
# ask-community
a
Hello, several times already I met a problem when a task with timeout never times out, actually. Python process that is orchestrated by Prefect is terminated sometimes by the GPU (something like OOM, not important now). I use @task(state_handlers=failure_notifiers, nout=2, timeout=60 * 60 * 4) decorator to set a timeout of 4 hours. In the Cloud dashboard, there is no mention of the timeout that I ordered for that task. Is the absence of timeout info in the dashboard expected, or my timeout is not being set, somehow?
k
Hey @Anatoly Alekseev, is the timeout taking effect? I think you would see it in the logs? Like a
TimedOutError
a
No, not taking at all. Task is hanging in Running state for days...
k
Gotcha, is the execution happening on different hardware than the flow? Like are you using a cluster or another VM?
What Executor are you using?
a
No no, just a local executor and a single server: with Flow( name="Ежедневное переобучение моделей", schedule=IntervalSchedule( start_date=pendulum.datetime(2019, 1, 1, 2, 0, 0, tz="Europe/Moscow"), interval=timedelta(days=1), ), storage=Local(add_default_labels=True), run_config=LocalRun(working_dir=WORKING_DIR) )
k
Got it. Will ask the team about it
👍 1
z
Hi! Can you run the flow with debug level logs? Perhaps set the timeout to a couple seconds so we can see if it fails quickly.
🙌 1
I'd also note that if your flow is killed by the machine, we can't enforce a timeout because the process that checks for a timeout is also likely killed. We generally recommend using Flow "did not finish" SLAs for this which can be setup under "Automations" in the UI. https://docs.prefect.io/orchestration/concepts/automations.html#automations
🙌 1
a
Got it, that should be the reason. Need to buy a paid plan! Thank you guys so much for looking into it. However, it would be amazing if timing out (or maybe failing) could occur even if the process was killed )