<@ULVA73B9P> I'm self-hosting prefect server and r...
# ask-marvin
s
@Marvin I'm self-hosting prefect server and run into weird failures: I run a background task which starts a flow running other tasks. All sub-tasks get into "COMPLETED" state, but I still get this issue in the flow and task that are set to "FAILED": Finished in state TimedOut("Flow run failed due to timeout: TimeoutError('Task run e01c9aed-7b96-4b3d-969a-f6e2939a5408 did not complete within None seconds')", type=FAILED) The task run pointed by the id here is set a "COMPLETED" and produces no error. How can I debug that ? how can I fix ?
m
thought for 242.1 seconds
Thanks for the details — this looks like a known edge case in Prefect 3.4.14 with background tasks and awaiting results. What you’re seeing: - Flow run ends in TimedOut with “Task run … did not complete within None seconds” - The referenced task run shows COMPLETED in the UI/logs Known issue - There’s an open report that matches your symptoms when using
.delay()
and then calling `.result()`/`.result_async()` before the background task has fully started: Issue #18775 - A fix was merged to clean up the confusing “None seconds” message and improve behavior around waiting for results: PR #18776 - This landed after 3.4.14. Please upgrade to the latest 3.4.x (3.4.15+) and re-test. Release notes: 3.4.14 → 3.4.15 Recommended next steps 1) Upgrade Prefect - Upgrade to 3.4.15 or newer and try the same run again. This should at least give clearer error messages; in many cases it resolves the premature timeout you’re hitting. 2) Confirm the await pattern - Make sure you’re only using the async APIs when inside an async flow, e.g.:
Copy code
# inside an async flow
futures = [my_task.with_options(name=f"t-{i}").delay(i) for i in range(n)]
results = await asyncio.gather(*(f.result_async() for f in futures))
- Avoid mixing
.result()
(sync) with
.result_async()
(async) in the same flow. 3) Try the “wait, then resolve” pattern (workaround) If you still see an issue after upgrading, try waiting for states first, then resolving results:
Copy code
from prefect.futures import wait, resolve_futures_to_results

futures = [my_task.with_options(name=f"t-{i}").delay(i) for i in range(n)]
done, not_done = await wait(futures)              # awaits completion of all futures
results = await resolve_futures_to_results(done)  # pulls the returned values
This avoids any race between task startup and the result call. 4) Add temporary instrumentation - Log each future’s state before awaiting results:
Copy code
for f in futures:
    s = await f.state_async()
    print("Future", f.task_run_id, "state:", s.type)
- If you see COMPLETED here but still get a timeout from
result_async()
, that strongly suggests the bug referenced above. 5) Double-check your worker setup - Background tasks must be served by a task worker. Ensure the worker is running and connected when you call
.delay()
, and that it’s serving the same task objects you’re delaying (dynamic names via
.with_options(name=...)
are fine). - Turn on debug logs for both flow and worker while reproducing:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
prefect version
Optional (self-hosted server) - If you notice odd ordering in events or states in the UI, consider setting
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=true
on your server (documented recently: PR #18774). This doesn’t directly cause the timeout, but can help with consistency in event/state displays. If you can share a minimal repro (your flow skeleton showing the
.delay()
+
asyncio.gather(*(f.result_async()))
pattern) I can try to reproduce exactly. Also include: - Prefect version after upgrade (
prefect version
) - How you start your task worker - Any flow/task
timeout_seconds
settings, if used Links for reference - Background tasks docs: Background tasks - Issue: Task run did not complete within None seconds - Fix PR: Fix confusing “None seconds” timeout message - Release notes 3.4.15: compare 3.4.14 → 3.4.15