https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> why is my task not retried at timeout...
# ask-marvin
o
@Marvin why is my task not retried at timeout although i defined retries? @task( retries=50, #effective endless retry_delay_seconds=two_scale_delays, retry_jitter_factor=0.25, retry_condition_fn=retry_download_slice, timeout_seconds=3600, #retry task taking longer than one hour ) Finished in state TimedOut("Flow run failed due to timeout: TaskRunTimeoutError('Scope timed out after 3600.0 second(s).')", type=FAILED) 085122 AM prefect.flow_runs Flow run failed due to timeout: TaskRunTimeoutError('Scope timed out after 3600.0 second(s).') 085122 AM prefect.flow_runs Finished in state TimedOut('Task run exceeded timeout of 3600.0 second(s)', type=FAILED) 085122 AM download_interval-466 prefect.task_runs Task run exceeded timeout of 3600.0 second(s)
m
✅ thought for 157.3 seconds
Most likely: your 
retry_condition_fn
is preventing the retry. In Prefect 3.x, a task timeout can be retried if: - 
retries > 0
- the retry condition allows it - the flow itself hasn’t timed out/stopped first What you’re seeing:
Copy code
Finished in state TimedOut('Task run exceeded timeout of 3600.0 second(s)', type=FAILED)
...
Flow run failed due to timeout: TaskRunTimeoutError(...)
does not mean retries are impossible. It means the task ended in a failed timeout state, and Prefect did not transition it to 
AwaitingRetry
/ 
Retrying
. The main reasons are: 1. 
retry_condition_fn
returned 
False
for the timeout Your function gets the failed state and decides whether to retry. If it inspects the exception and excludes 
TaskRunTimeoutError
, Prefect will fail the task immediately. A timeout-safe version looks like:
Copy code
python
from prefect.task_engine import TaskRunTimeoutError

def retry_download_slice(task, task_run, state):
    try:
        state.result()
    except TaskRunTimeoutError:
        return True
    except Exception:
        return True
    return True
If you only want to retry some exceptions, make sure timeout is included explicitly. 2. 
retry_condition_fn
itself is erroring If that function raises an exception while evaluating the failed state, Prefect will generally skip retrying. For example, this is fragile:
Copy code
python
def retry_download_slice(task, task_run, state):
    exc = state.result()
    return exc.status_code != 404
because 
state.result()
raises the underlying exception instead of returning a neat object. Safer pattern:
Copy code
python
from prefect.task_engine import TaskRunTimeoutError

def retry_download_slice(task, task_run, state):
    try:
        state.result()
    except TaskRunTimeoutError:
        return True
    except SomePermanentError:
        return False
    except Exception:
        return True
    return True
3. A flow-level timeout may be confusing the logs Your log includes:
Copy code
prefect.flow_runs
Flow run failed due to timeout: TaskRunTimeoutError(...)
That message can be misleading. It may just mean the flow failed because the task timed out. But if your flow also has a timeout, that can stop retries entirely. So check whether your flow decorator also has something like:
Copy code
python
@flow(timeout_seconds=...)
If the flow times out before the retry delay / next attempt, the task will not retry. 4. Sync blocking work can make timeout behavior tricky If this task is a synchronous task doing blocking I/O or long-running C/library calls, timeout handling may not interrupt it as cleanly as expected depending on execution context. But in your case, since the task did end as 
TimedOut
, the more likely issue is still the retry condition. What I’d try first Temporarily remove 
retry_condition_fn
and see whether the timeout retries:
Copy code
python
@task(
    retries=3,
    retry_delay_seconds=[10, 30, 60],
    timeout_seconds=3600,
)
def download_interval(...):
    ...
If that retries, then the problem is definitely inside 
retry_download_slice
. Then reintroduce a minimal condition:
Copy code
python
from prefect.task_engine import TaskRunTimeoutError

def retry_download_slice(task, task_run, state):
    try:
        state.result()
    except TaskRunTimeoutError:
        return True
    except Exception:
        return True
    return True
Recommended debugging step Add logging inside `retry_download_slice`:
Copy code
python
from prefect.logging import get_run_logger
from prefect.task_engine import TaskRunTimeoutError

def retry_download_slice(task, task_run, state):
    logger = get_run_logger()
    try:
        state.result()
    except TaskRunTimeoutError as exc:
        <http://logger.info|logger.info>(f"Retrying after timeout: {exc}")
        return True
    except Exception as exc:
        <http://logger.info|logger.info>(f"Retry condition saw exception: {type(exc).__name__}: {exc}")
        return True
    return True
If you want, paste your 
retry_download_slice
function here and I can tell you exactly why the timeout is not being retried.
3 Views
Open in Slack
PreviousNext
Powered by