<@ULVA73B9P> when setting up retries for a task, w...
# ask-marvin
m
@Marvin when setting up retries for a task, what does the task need to return or raise for it trigger. Specifically working with API requests to external services
m
thought for 147.7 seconds
Great question. In Prefect 3.x, a task will retry only if it ends in a Failed state. That happens when: - The task raises an exception (most common) - The task explicitly returns a
prefect.states.Failed(...)
state What will NOT trigger a retry: - Returning any normal value (including
None
) - Swallowing errors in a try/except without re-raising Tips for API calls to external services: - Let your HTTP client raise on error (e.g.,
response.raise_for_status()
), or re-raise selectively if you catch exceptions - Optionally use
retry_condition_fn
to control which failures should retry (e.g., retry on 5xx/429, skip on 4xx like 401/404) - Use backoff and jitter to avoid hammering the service Example pattern with httpx:
Copy code
import httpx
from prefect import task, flow
from prefect.states import Failed
from prefect.tasks import exponential_backoff

def should_retry(task, task_run, failed_state) -> bool:
    # Inspect the exception from the failed state
    try:
        failed_state.result()  # raises the original exception
    except httpx.HTTPStatusError as exc:
        status = exc.response.status_code
        # Retry on 5xx and 429 (rate limits), skip on common non-retryable 4xx
        return status >= 500 or status == 429
    except httpx.ConnectError:
        return True  # transient network issue
    except Exception:
        return True  # default: retry other unexpected errors

@task(
    retries=5,
    retry_delay_seconds=exponential_backoff(backoff_factor=2),  # 1s,2s,4s,8s,16s...
    retry_jitter_factor=0.5,
    retry_condition_fn=should_retry,
)
def fetch_json(url: str):
    r = httpx.get(url, timeout=10)
    r.raise_for_status()          # raising here triggers a retry
    return r.json()

@flow
def run():
    return fetch_json("<https://api.example.com/data>")
If you prefer to fail explicitly (instead of raising), you can return a Failed state:
Copy code
from prefect.states import Failed

@task(retries=3)
def maybe_fail(flag: bool):
    if flag:
        return Failed(message="Intentional failure")  # triggers retry
    return "ok"
Notes: - Timeouts and any unhandled exceptions count as failures and will trigger retries (if configured). - There is no “retry signal” to raise; you either raise an exception or return a Failed state. -
retry_condition_fn
can be sync or async and receives
(task, task_run, state)
. Docs: - Retries guide: Prefect retries how-to If you can share which client you’re using and which HTTP statuses you want to retry vs skip, I can tailor the
retry_condition_fn
for your case.