https://prefect.io
Join Slack
Greetings, friends! Anyone have experience with adding timeouts to the child tasks spawn from mappin...
# ask-community
c
Greetings, friends! Anyone have experience with adding timeouts to the child tasks spawn from mapping? With a single task, I have no problem adding timeout arg to @task decorator. But when I do the same for a task that is mapped, there seems to be no effect.
👀 1
k
Hi @Chris Leber! I’ll look into this for you.
c
Thanks @Kevin Kho!! I very much appreciate it.
k
I have this code snippet and it seems to work:
Copy code
from prefect import Flow, task
import time

@task(timeout=2)
def abc(x):
    time.sleep(3)
    return x

with Flow(
    "mapped-failures",
) as flow:
    map_results = abc.map([1,2,3,4])

flow.run()
Is yours failing on Cloud/Server? or also on LocalExecution? Is this representative of what you have?
c
Failing on Cloud, and yes representative of what I am doing 🤔 quite perplexing
I wonder if it has to do with the nature of the task, as it seems to be causing all sorts of strangeness. The task involves using pycurl for downloading files from a database. Sometimes the files are available for downloading immediately, sometimes they are 'pending' as they need to be retrieved from a tape archive (can take up to 24 hours). Other strangeness: I cancelled a few flows that were getting stuck on one of these 'pending' files - hours later, the flows do not report that they have been "cancelled". Instead, they still report that they are "cancelling", and a look at their tasks shows that the task in question is indeed still "running".
k
Let me test this on cloud real quick
So cloud is working on my end with the LocalDaskExecutor. Are you on a DaskExecutor?
c
Currently not explicitly selecting an executor
So whatever the default executor is
Trying LocalDaskExecutor currently
k
This is odd. I’ll probably have to look at it a bit later
c
LocalDaskExecutor allowed timeouts to occur! What is the default Executor? Still strange that the default did not allow
k
LocalExecutor, which runs things sequentially
Maybe it didn’t feel like a timeout because of the sequential execution of the mapped tasks?
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by