https://prefect.io logo
Title
k

Kaz

04/05/2020, 7:23 PM
Hey guys. Big fan of the product, it’s really cut down on our manual ETL work load. 1 issue: setting the time out in a
ShellTask
doesn’t seem to work for me. I have an S3 sync that occasionally hang (working to figure that out), and as a temporary solution I was hoping to use a timeout + retry. This has solved the issue in the past. I’ve set the timeout to 600 (600 seconds, 10 minutes), and yet I’m getting a task that runs for 6 hours+ when the shell task hangs. On Cloud (Scheduler tier), I can see that my retry and retry delay parameters are set, but I’m not sure if there’s a way to check whether a timeout has been configured on a task. It’s a fairly simple task (top level, only 1 dependency, unmapped), and everything works just fine if I manually restart the task on the Cloud UI (I even see the retry count increment). I’m wondering if there are any additional inputs I needed to configure other than timeout + retry in order to get the timeout to work, or if this is an issue other people have seen before?. Thanks!
c

Chris White

04/05/2020, 7:38 PM
Hi Kaz! Great question; timeouts are a tricky piece of functionality in the Prefect world. Currently timeouts operate on a “best effort” basis and are implemented as general purpose utility functions which are applied across all tasks independent of implementation. Because of this, certain types of tasks don’t play nicely with timeouts (because ultimately Prefect doesn’t know what the task is doing under the hood and can only halt / cleanup so much). We haven’t yet implemented task-specific timeout logic, but in the case of the
ShellTask
we certainly could. If this is important for your workflow we should open an issue on the repo to look into fixing this!
k

Kaz

04/05/2020, 9:13 PM
Understood. Happy to open an issue on GitHub to get this addressed. For now, would the best solution be to define my own task with custom logic to raise a timeout error?
c

Chris White

04/05/2020, 10:03 PM
If you wouldn’t mind opening an issue that would be great! Yea in the meantime, if you use the
ShellTask
code but then replace these lines: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/shell.py#L95-L104 with this one line:
sub_process.wait(timeout=600)
you should be good