Hey, i have a task that has ran successfully in the past, but now it gets stuck on ’Task ‘X’: Starting task run...’. All other tasks run fine, and if i manually kill the job and restart it, it runs fine again. I’m running on EKS, so its spinning up a pod to run this task. Any idea’s for debugging? The task just doesn’t seem to start

Can you share a bit more? What is this task doing? Could it be some issue with unclosed files or DB connections? If you could share either your flow or a minimal reproducible example, I could check if I can reproduce and see what might be the issue there. Is this some memory-intensive or long-running task?

It is connecting and downloading a file from the FTP. It could be related due to the connection, but it only takes 2 seconds to run normally. And haven’t had this problem on restarts.

There are 2 options for how you could approach it: 1) You could set a timeout in seconds:

@task(timeout=120)

2) A bit more effort but more reliable: you could separate out this "problematic" task into a separate flow and call it within the "parent" flow using

create_flow_run

task. This way, you could set a time-based SLA Automation on the flow that does this FTP processing and SLA is very reliable so you could this way ensure that any time this flow takes longer than say 3 minutes, the flow run should be canceled.

Do the pod logs show anything?

SLA automation: and here is an example with the syntax for subflow https://discourse.prefect.io/t/how-can-i-create-a-subflow-and-block-until-it-s-completed/94/2

Awesome, thanks! Will read into this. Checking pod logs is a good idea, I unfortunately deleted the pods as part of the shutdown process - otherwise it was just stalled for hours. But will check the pod logs next time I run into this