Nicholas Pini
08/06/2026, 3:47 PMwait() on a list of tasks and one of them fails? Will the flow fail too?Marvin
08/06/2026, 3:47 PMMarvin
08/06/2026, 3:49 PMwait() on a list of task futures does not itself raise if one task fails.
What happens:
- wait(futures) waits until the futures reach a terminal state
- a failed task is still considered done
- the flow does not automatically fail just because wait() saw a failed future
The flow will typically fail only if you later do something like:
- future.result() on the failed future, or
- otherwise explicitly raise based on its state
Example:
python
from prefect import flow, task
from prefect.futures import wait
@task
def maybe_fail(x):
if x == 2:
raise ValueError("boom")
return x * 10
@flow
def my_flow():
futures = [maybe_fail.submit(i) for i in range(4)]
done, not_done = wait(futures)
print(f"done={len(done)}, not_done={len(not_done)}")
for f in futures:
print(f.state)
# This is where failure would surface:
return [f.result() for f in futures]
In that flow:
- wait(futures) completes normally
- one future will have a failed state
- the flow fails when [f.result() for f in futures] hits the failed one
If you want to handle partial failures gracefully, use one of these patterns:
python
results = []
for f in futures:
if f.state.is_failed():
results.append(None)
else:
results.append(f.result())
or:
python
results = [f.result(raise_on_failure=False) for f in futures]
With raise_on_failure=False, failed task results come back as exception objects instead of being raised.
So the short answer is:
- Will wait() raise? No
- Will the flow fail too? Not from wait() alone
- When does it fail? Usually when you access the failed future’s result
If you want, I can also show the exact behavior of wait() vs as_completed() vs gather() in Prefect 3.Nicholas Pini
08/06/2026, 4:32 PMMarvin
08/06/2026, 4:32 PMMarvin
08/06/2026, 4:33 PMtimeout=None means wait indefinitely.
It does not return immediately.
So for `wait_for_flow_run(...)`:
- default is 10800 seconds (3 hours)
- timeout=None disables the timeout
- it waits until the flow run reaches a final state
Example:
python
await wait_for_flow_run(flow_run_id, timeout=None)
That will block forever unless the flow run eventually finishes, fails, or is cancelled.
One caveat: if the flow run gets stuck and never reaches a final state, your caller will also wait forever.
If you want, I can also show the exact exception behavior for a numeric timeout.Nicholas Pini
08/06/2026, 4:35 PMwith LOCK:
flow_run = await arun_deployment(
name=flow_name,
flow_run_name=flow_run_name,
parameters=dict(**custom_args, s3_output_folder=s3_output_path.as_uri(), index_range=index_range),
job_variables=jv,
)
append_uuid(flow_run.id)
await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
assert flow_run.state is not None
try:
await flow_run.state.aresult()
except Exception as e:
logger.error("pet-wall-aspect-wall-height failed")
raise e
<http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
why did this code fail with
Task run failed with exception: UnfinishedRun('Run is in SCHEDULED state, its result is not available.')
Traceback (most recent call last):
File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1596, in run_context
yield self
File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1675, in run_task_async
await engine.call_task_fn(txn)
File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1613, in call_task_fn
result = await call_with_parameters(self.task.fn, parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/.venv/lib/python3.12/site-packages/pet_wrapper/prepare_tiles.py", line 2571, in prepare_tiles
raise e
File "/opt/.venv/lib/python3.12/site-packages/pet_wrapper/prepare_tiles.py", line 2568, in prepare_tiles
await flow_run.state.aresult()
File "/opt/.venv/lib/python3.12/site-packages/prefect/client/schemas/objects.py", line 286, in aresult
return await get_state_result(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/.venv/lib/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
return await _get_state_result(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/.venv/lib/python3.12/site-packages/prefect/states.py", line 150, in _get_state_result
raise UnfinishedRun(
prefect.exceptions.UnfinishedRun: Run is in SCHEDULED state, its result is not available.
it's like it has never waited for the run to finishMarvin
08/06/2026, 4:35 PMMarvin
08/06/2026, 4:37 PMwait_for_flow_run() waited, but you ignored the updated FlowRun object it returns.
Your code does this:
python
flow_run = await arun_deployment(...)
await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
await flow_run.state.aresult()
But flow_run there is still the original object returned by arun_deployment, and that object can still have the old state like SCHEDULED.
wait_for_flow_run() does not mutate that object in place. It returns a fresh FlowRun fetched from the API once the run reaches a final state.
So this is the correct pattern:
python
flow_run = await arun_deployment(
name=flow_name,
flow_run_name=flow_run_name,
parameters=dict(
**custom_args,
s3_output_folder=s3_output_path.as_uri(),
index_range=index_range,
),
job_variables=jv,
)
append_uuid(flow_run.id)
flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
assert flow_run.state is not None
try:
await flow_run.state.aresult()
except Exception as e:
logger.error("pet-wall-aspect-wall-height failed")
raise
<http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
Why you saw SCHEDULED specifically
- arun_deployment() initially creates the run in a scheduled state
- that returned Python object is just a snapshot
- the actual run may later move to RUNNING, COMPLETED, FAILED, etc.
- your original flow_run variable does not auto-refresh
- flow_run.state.aresult() checks that stale state object, sees SCHEDULED, and raises UnfinishedRun
So the failure message:
python
UnfinishedRun('Run is in SCHEDULED state, its result is not available.')
does not mean wait_for_flow_run() failed to wait. It means you called aresult() on the stale pre-wait object.
One more thing: if you're already using wait_for_flow_run(..., timeout=None), you probably don't need both:
- arun_deployment(...)
- then wait_for_flow_run(...)
Depending on your usage, arun_deployment may already support waiting behavior, but the safest fix for your current code is simply to reassign:
python
flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
If you want, I can also help simplify this whole block into the cleanest Prefect 3 pattern for “run deployment and raise if child flow failed.”Nicholas Pini
08/06/2026, 4:38 PMMarvin
08/06/2026, 4:38 PMMarvin
08/06/2026, 4:41 PMpython
await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
then wait_for_flow_run() did not behave like an infinite blocker in that execution.
Based on the implementation, the most likely explanation is:
wait_for_flow_run() relies on an event subscription, and that subscription can terminate early.
Most important case:
- if the WebSocket/event stream closes cleanly
- Prefect's event subscriber can stop iterating
- wait_for_flow_run() can exit without the target run reaching a final state
So in practice, timeout=None means “no timeout on the wait scope,” but not necessarily “guaranteed to block forever no matter what happens to the event stream.”
That matches your observation:
- child flow still unfinished
- parent continued
- stale flow_run.state still said SCHEDULED
- aresult() then raised UnfinishedRun
What I’d recommend
Instead of trusting wait_for_flow_run() alone for this case, use a polling-based wait or hybrid wait.
Example polling approach:
python
import asyncio
from prefect.client.orchestration import get_client
async def wait_until_final(flow_run_id, poll_interval=5):
async with get_client() as client:
while True:
flow_run = await client.read_flow_run(flow_run_id)
if flow_run.state is not None and flow_run.state.is_final():
return flow_run
await asyncio.sleep(poll_interval)
Then:
python
with LOCK:
flow_run = await arun_deployment(
name=flow_name,
flow_run_name=flow_run_name,
parameters=dict(
**custom_args,
s3_output_folder=s3_output_path.as_uri(),
index_range=index_range,
),
job_variables=jv,
)
append_uuid(flow_run.id)
flow_run = await wait_until_final(flow_run.id)
try:
await flow_run.state.aresult()
except Exception:
logger.error("pet-wall-aspect-wall-height failed")
raise
<http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
Why this is safer
- it does not depend on the events/WebSocket channel staying alive
- every loop iteration fetches fresh state from the API
- if the run is still SCHEDULED or RUNNING, it keeps waiting
Important nuance
There are really two separate issues here:
1. you must reassign the returned FlowRun
2. in your case, wait_for_flow_run() itself also seems to have returned prematurely
Given your symptom, I think both are true.
If you want, I can help you turn your exact block into a hardened helper like run_deployment_and_wait() with:
- polling
- optional timeout
- child-failure propagation
- better logging of stuck SCHEDULED runsNicholas Pini
08/06/2026, 4:44 PMMarvin
08/06/2026, 4:44 PMMarvin
08/06/2026, 4:44 PMwait_for_flow_run() is:
- convenience
- low-latency waiting
- event-driven monitoring instead of polling the API repeatedly
In the happy path, it’s nice because it:
- subscribes to flow-run state change events
- wakes up as soon as the run reaches a final state
- avoids constant read_flow_run(...) calls
So conceptually it exists to be a lightweight “wait until this flow run finishes” helper.
But the tradeoff is that it depends on the event stream infrastructure working end-to-end:
- client event subscriber
- websocket connection
- server event delivery
- no disconnects or subscriber edge cases
That makes it less robust than brute-force polling for critical orchestration.
So I’d frame it like this:
- Good for: normal interactive waiting, simple scripts, happy-path orchestration
- Less ideal for: mission-critical parent/child orchestration where “must not proceed until final state” is a hard requirement
For that stricter use case, polling is often the more reliable primitive:
python
async def wait_until_final(flow_run_id, poll_interval=5):
async with get_client() as client:
while True:
flow_run = await client.read_flow_run(flow_run_id)
if flow_run.state and flow_run.state.is_final():
return flow_run
await asyncio.sleep(poll_interval)
So the point of wait_for_flow_run() is not “guaranteed strongest synchronization primitive,” but “efficient event-based waiting helper.”
Given your use case, I would not rely on it alone. I’d use:
- polling only, or
- event wait with polling fallback
If you want, I can give you a production-safe wait_for_flow_run_strict() helper you can drop into your codebase.Nicholas Pini
08/06/2026, 4:45 PMMarvin
08/06/2026, 4:45 PMMarvin
08/06/2026, 4:45 PMwait_for_flow_run() was waiting on the event stream, and that stream ended before your child flow reached a final state.
So the function felt like “wait until done,” but internally it is closer to:
- subscribe to flow-run events
- return when a final-state event is seen
- but if the subscription stops unexpectedly, it may stop waiting
For your case, the strongest explanation is:
1. arun_deployment(...) created the child run
2. child run stayed non-final (SCHEDULED / not finished)
3. wait_for_flow_run(flow_run.id, timeout=None, log_states=True) relied on Prefect events
4. that event subscription likely disconnected or terminated early
5. your code resumed
6. you then checked the old flow_run.state, which was still SCHEDULED
7. aresult() raised UnfinishedRun
So it didn't work for you because the waiting mechanism is event-driven, not a strict polling loop, and event delivery/connection appears to have failed in your run.
A few plausible concrete causes:
- websocket/event subscriber closed cleanly
- network interruption between your runner and Prefect API
- server-side event stream interruption
- a bug/edge case in the subscriber loop
- the child run never emitted the expected final-state event to that subscriber session
What to do next
For your use case, I’d treat wait_for_flow_run() as best-effort, not strict. Use polling instead:
python
import asyncio
from prefect.client.orchestration import get_client
async def wait_until_final(flow_run_id, poll_interval=5):
async with get_client() as client:
while True:
flow_run = await client.read_flow_run(flow_run_id)
state = flow_run.state
if state is not None and state.is_final():
return flow_run
await asyncio.sleep(poll_interval)
Then:
python
flow_run = await arun_deployment(...)
flow_run = await wait_until_final(flow_run.id)
await flow_run.state.aresult()
If you want to confirm this was an event-stream problem, add debug logging around the events client and compare timestamps:
python
import logging
logging.getLogger("prefect.events.clients").setLevel(logging.DEBUG)
So the short answer is:
it likely didn’t work because the event subscription ended before the child run finished, and wait_for_flow_run() is built on that subscription rather than guaranteed polling.
If you want, I can help you build a drop-in replacement with:
- strict polling
- timeout support
- logging of state transitions
- detection of “stuck in SCHEDULED” runsNicholas Pini
08/06/2026, 4:48 PMMarvin
08/06/2026, 4:48 PMMarvin
08/06/2026, 4:48 PMwait_for_flow_run() depends on a live client-side event stream, so failure does not require “the server is broken.” It can happen even when the server is generally healthy.
Possible reasons:
- the websocket closed normally and the client treated that as end-of-stream
- a transient network hiccup between your runner and Prefect API
- a proxy / load balancer / ingress idle timeout
- container/job lifecycle behavior in your execution environment
- an edge case in Prefect’s event subscriber handling
- the subscriber missed the event and had no polling fallback
So “server looks fine” does not rule this out.
In your environment, I’d especially suspect infrastructure between the runner and API:
- AWS CodeBuild / CI jobs
- NAT/proxy behavior
- load balancer websocket handling
- idle connection timeouts
Because event-based waiting is more sensitive to those than normal API polling.
Why this is confusing
Most Prefect operations are request/response HTTP:
- create run
- read run
- update state
Those can all work perfectly.
But wait_for_flow_run() is different:
- it opens a longer-lived subscription
- that is inherently more fragile than short API calls
So you can have:
- healthy Prefect server
- healthy database
- normal UI/API behavior
- and still a broken or interrupted event wait in one client session
How to verify
1. Turn on debug logs for the events client:
python
import logging
logging.getLogger("prefect.events.clients").setLevel(logging.DEBUG)
2. Log around the wait:
python
<http://logger.info|logger.info>(f"waiting for child flow run {flow_run.id}")
updated_flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
<http://logger.info|logger.info>(f"wait returned with state={updated_flow_run.state}")
3. If possible, immediately compare with a fresh API read:
python
async with get_client() as client:
fresh = await client.read_flow_run(flow_run.id)
<http://logger.info|logger.info>(f"fresh state after wait: {fresh.state}")
If wait_for_flow_run() returns but a fresh read shows non-final state, that strongly confirms the wait mechanism exited early.
Bottom line
You do not need a broken server for this to happen. A healthy server plus:
- websocket close
- networking middlebox timeout
- subscriber bug
is enough.
For anything critical, I’d recommend polling as the source of truth.
If you want, I can give you a small diagnostic wrapper that logs:
- event wait start/end
- returned state
- immediate fresh-read state
- elapsed time
so you can prove exactly where it breaks.Nicholas Pini
08/06/2026, 5:08 PMasync def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.state.aresult()Marvin
08/06/2026, 5:08 PMMarvin
08/06/2026, 5:08 PMwait_for_flow_run() by itself, but the interaction between:
- run_deployment(..., timeout=0)
- child runs tracked in a global list
- parent on_failure hook canceling those child runs
- tasks awaiting child completion
- as_completed(tasks) in the parent flow
What your pattern is effectively doing
1. A task starts a child deployment with timeout=0
- so it returns immediately
- child flow run may still be SCHEDULED or RUNNING
2. The task stores the child run id globally
3. The task then waits for that child run to finish
4. If one sibling task fails, the parent flow can fail
5. Your parent on_failure hook runs and cancels all child runs in the global list
6. Other tasks that were waiting on child runs may now see cancellation / non-final / stale state behavior
That means you have real concurrency between:
- child runs progressing
- parent flow failure
- cancellation hook firing
- task futures completing/failing
- your wait_for_tasks() loop consuming task results
That can produce confusing states.
The most important problem in your code is here:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.state.aresult()
I would not rely on task.state.aresult() on the future's state attribute like that.
Why:
- task.state may be stale
- as_completed(tasks) tells you the future completed, but the state attribute you read may not be the freshest representation you expect
- the normal future interface is await task.result() or task.result(), not task.state.aresult()
You should prefer:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.result()
or if async result access isn't available on that future type in your context, then at least:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
state = await task.wait()
await task.result()
But the cleanest is usually just:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.result()
Why this matters
PrefectFuture.state is not the same thing as “always current authoritative task state from the API.”
When you do:
python
await task.state.aresult()
you're calling aresult() on whatever State object is attached there at that moment. If that state is stale or non-final, you can get misleading exceptions.
By contrast, task.result() is the future abstraction meant to surface completion/failure properly.
Another likely issue: parent failure causes child cancellation while siblings are still waiting
Suppose:
- one task fails
- parent flow begins failing
- on_failure hook cancels child runs
- another task is still in wait_for_flow_run(child_id, timeout=None)
Now that waiting task may:
- return due to event-stream weirdness
- see child cancellation
- see stale flow-run state
- fail while trying to read result
So your observed behavior may be a consequence of cancellation orchestration, not just a standalone wait bug.
Global list is also risky
A global mutable list of run IDs can be problematic in flows/tasks because:
- concurrent tasks mutate shared state
- retries can duplicate IDs
- task runner/process boundaries may make globals unreliable
- failure hooks may run with partial or stale contents
Safer patterns are:
- return child run IDs from tasks and collect them in the flow
- or persist them in Prefect artifacts / external storage / a thread-safe structure
- or model orchestration so the flow owns child-run bookkeeping explicitly
What I would change first
1. Replace this:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.state.aresult()
with this:
```pythonMarvin
08/06/2026, 5:08 PM2. Do not depend on `wait_for_flow_run()` as the only child-wait primitive for critical logic; use polling.
3. Avoid reading `.state.aresult()` on stale flow-run objects; refresh first or use the future/result abstraction.
4. Reconsider global bookkeeping for child run IDs.
A more robust task pattern would look like:
```python
import asyncio
from prefect import task
from prefect.client.orchestration import get_client
from prefect.deployments.flow_runs import run_deployment
async def wait_until_final(flow_run_id, poll_interval=5):
async with get_client() as client:
while True:
flow_run = await client.read_flow_run(flow_run_id)
if flow_run.state and flow_run.state.is_final():
return flow_run
await asyncio.sleep(poll_interval)
@task
async def launch_and_wait(...):
flow_run = await run_deployment(
name="my-deployment",
parameters={...},
timeout=0,
)
append_uuid(flow_run.id)
flow_run = await wait_until_final(flow_run.id)
await flow_run.state.aresult()
return flow_run.id
Then:
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.result()
Bottom line
I think your issue is likely a combination of:
- using wait_for_flow_run() in a critical synchronization path
- then reading stale .state
- plus parent failure/cancellation racing with child waits
- plus using task.state.aresult() instead of the future’s result() interface
If you want, I can help you redesign this into a safer parent/child orchestration pattern that preserves:
- “launch many child deployments”
- “cancel all children if parent fails”
- “surface the first child failure cleanly”
without relying on globals and fragile event waits.Nicholas Pini
08/06/2026, 5:56 PMMarvin
08/06/2026, 5:56 PMMarvin
08/06/2026, 5:58 PMprefect.futures.as_completed() and call .result() on each future as it completes.
That gives you fail-fast behavior: as soon as the first completed task is in a failed state, .result() raises immediately, and you do not wait for the rest.
Pattern:
python
from prefect.futures import as_completed
async def wait_for_tasks_fail_fast(tasks):
for task in as_completed(tasks):
await task.result()
If your futures are not awaitable with await task.result() in your context, use:
python
from prefect.futures import as_completed
def wait_for_tasks_fail_fast(tasks):
for task in as_completed(tasks):
task.result()
The important part is:
- as_completed(tasks) yields futures in completion order
- task.result() raises on failure by default
- so the first failed task to finish stops the loop immediately
This is exactly what you want, and it’s different from:
- wait(tasks) → waits for all tasks to finish
- iterating the original list and calling .result() → can block on an earlier slow task and miss a later fast failure
So your fail-fast version should be:
python
from prefect.futures import as_completed, PrefectFuture
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
await task.result()
If you also want to cancel sibling child flow runs as soon as one fails, I can show how to combine this with your global UUID cancellation logic so you fail fast and actively stop the remaining work.Nicholas Pini
08/06/2026, 5:59 PMMarvin
08/06/2026, 6:00 PMMarvin
08/06/2026, 6:00 PM.result() is usually not an async coroutine, so await task.result() is wrong.
Use:
python
from prefect.futures import as_completed, PrefectFuture
async def wait_for_tasks(tasks: list[PrefectFuture]):
for task in as_completed(tasks):
task.result()
Not:
python
await task.result()
Why this works:
- as_completed(tasks) only yields futures once they are done
- task.result() then immediately returns the result or raises the task exception
- if one task failed, task.result() raises right there
- so this is still fail-fast
So the correct fail-fast pattern is:
python
from prefect.futures import as_completed
async def wait_for_tasks_fail_fast(tasks):
for task in as_completed(tasks):
task.result()
If you want to preserve the async function signature, that’s fine — just don’t await the .result() call.
If you'd like, I can also show you how to cancel the remaining tasks / child deployments immediately when one of those task.result() calls raises.Nicholas Pini
08/06/2026, 6:02 PMMarvin
08/06/2026, 6:02 PMMarvin
08/06/2026, 6:02 PMas_completed() loop in try/except, and when one task fails, cancel the remaining child flow runs before re-raising.
Something like this:
python
from prefect.futures import as_completed, PrefectFuture
from prefect.client.orchestration import get_client
# however you store these
CHILD_FLOW_RUN_IDS: set[str] = set()
async def cancel_child_flow_runs(flow_run_ids):
async with get_client() as client:
for flow_run_id in flow_run_ids:
try:
await client.cancel_flow_run(flow_run_id)
except Exception:
# optional: log and continue
pass
async def wait_for_tasks_fail_fast(tasks: list[PrefectFuture]):
try:
for task in as_completed(tasks):
task.result() # raises immediately if this task failed
except Exception:
await cancel_child_flow_runs(CHILD_FLOW_RUN_IDS)
raise
If your IDs are UUIDs instead of strings, that’s fine too.
A slightly safer version tracks only the runs that are still active:
python
from prefect.futures import as_completed, PrefectFuture
from prefect.client.orchestration import get_client
CHILD_FLOW_RUN_IDS: set = set()
async def cancel_child_flow_runs(flow_run_ids):
async with get_client() as client:
for flow_run_id in flow_run_ids:
try:
await client.cancel_flow_run(flow_run_id)
except Exception:
pass
async def wait_for_tasks_fail_fast(tasks: list[PrefectFuture]):
try:
for task in as_completed(tasks):
task.result()
except Exception:
await cancel_child_flow_runs(CHILD_FLOW_RUN_IDS)
raise
And if you want logging:
python
from prefect.futures import as_completed
from prefect.client.orchestration import get_client
async def cancel_child_flow_runs(flow_run_ids, logger):
async with get_client() as client:
for flow_run_id in flow_run_ids:
try:
<http://logger.info|logger.info>(f"Cancelling child flow run {flow_run_id}")
await client.cancel_flow_run(flow_run_id)
except Exception as exc:
logger.warning(f"Failed to cancel child flow run {flow_run_id}: {exc}")
async def wait_for_tasks_fail_fast(tasks, child_flow_run_ids, logger):
try:
for task in as_completed(tasks):
task.result()
except Exception as exc:
logger.error(f"A task failed, cancelling remaining child flow runs: {exc}")
await cancel_child_flow_runs(child_flow_run_ids, logger)
raise
A couple of notes:
- task.result() is the thing that raises the task failure
- as_completed() gives you fail-fast behavior based on completion order
- canceling child flow runs won’t automatically cancel already-finished ones, so it’s okay if some IDs are stale
- if multiple tasks can mutate the global set concurrently, use a lock or a copy before iterating
For example:
python
await cancel_child_flow_runs(set(CHILD_FLOW_RUN_IDS), logger)
If you want, I can help you make this fully race-safe with:
- a lock around the run-id set
- removing IDs after successful completion
- avoiding duplicate cancel attemptsNicholas Pini
08/07/2026, 8:23 AMclient.cancel_flow_run available in prefect 3.6.12?Marvin
08/07/2026, 8:23 AMMarvin
08/07/2026, 8:25 AMclient.cancel_flow_run is not available in Prefect 3.6.12.
The supported Python-client way is to set the flow run state to Cancelling via set_flow_run_state(...).
Example:
python
from prefect.client.orchestration import get_client
from prefect.states import Cancelling
async def cancel_flow_run(flow_run_id):
async with get_client() as client:
await client.set_flow_run_state(
flow_run_id,
Cancelling(message="Cancelled because a sibling task failed"),
)
So if you want to cancel remaining child flow runs on first failure, use that pattern instead of client.cancel_flow_run(...).
If helpful, your earlier helper should become:
python
from prefect.client.orchestration import get_client
from prefect.states import Cancelling
async def cancel_child_flow_runs(flow_run_ids):
async with get_client() as client:
for flow_run_id in flow_run_ids:
try:
await client.set_flow_run_state(
flow_run_id,
Cancelling(message="Cancelled because a sibling task failed"),
)
except Exception:
pass
Related docs:
- Cancel workflows
- Python client API
One important note: setting Cancelling requests cancellation; actual stopping depends on the worker/infrastructure picking that up.Nicholas Pini
08/07/2026, 10:30 AM.submit(wait_for=list_of_tasks), I get this LSP error:
1. No overloads for "submit" match the provided arguments
Argument types: (str, list[PrefectFuture[CoroutineType[Any, Any, None]]]) [reportCallIssue]
list_of_tasks is a list of PrefectFuture[CoroutineType[Any, Any, None]] , and when I run the code it works correctly. Why do I get the LSP error?Marvin
08/07/2026, 10:30 AMMarvin
08/07/2026, 10:33 AM.submit(wait_for=...) accepts your list and works
- but the type checker sees your async task futures as PrefectFuture[CoroutineType[Any, Any, None]]
- the submit() overloads don’t fully account for that case unless async-task typing is narrowed correctly
- so Pyright/Pylance reports “no overload matches,” even though Prefect executes it fine
In short: the LSP is complaining about Prefect’s type hints, not your actual code behavior.
Why specifically this happens with async tasks:
- for an async def task, the raw Python function returns a coroutine
- Prefect’s runtime unwraps that correctly
- but the static typing around Task.submit(...) can infer the future as wrapping Coroutine[...] instead of the resolved result type
- then `wait_for`’s overload no longer matches cleanly
So your list[PrefectFuture[CoroutineType[Any, Any, None]]] is the clue: that type is what the LSP doesn’t like.
Workarounds
1. Best workaround: pass return_state=False explicitly
python
next_task.submit("arg", wait_for=list_of_tasks, return_state=False)
That can help the type checker choose the intended overload.
2. Cast the list if you want to silence the warning cleanly
python
from typing import cast
from prefect.futures import PrefectFuture
deps = cast(list[PrefectFuture[None]], list_of_tasks)
next_task.submit("arg", wait_for=deps)
3. Use a targeted ignore if needed
python
next_task.submit("arg", wait_for=list_of_tasks) # type: ignore[arg-type]
My recommendation:
- if runtime behavior is correct, treat this as a Prefect typing issue
- use either explicit return_state=False or a cast(...)
So the short answer is:
you get the LSP error because Prefect’s submit() overload typing for async-task futures is imperfect, and your editor is enforcing the static types more strictly than runtime does.
If you want, I can help you write the least ugly cast/helper so this disappears throughout your codebase.Nicholas Pini
08/07/2026, 12:29 PMPrefectFuture[CoroutineType[Any, Any, None]] object? It has a task_run_id property but it is deprecated (I am using Prefect 3.6.12). I want to log a message each time a task in a list of tasks is completedMarvin
08/07/2026, 12:29 PMMarvin
08/07/2026, 12:31 PMfuture.task_name property on a PrefectFuture.
If you want the task run name, the supported way is:
- get the future’s task_run_id
- read the task run from the client
- use task_run.name
And importantly: if your future is a task future from .submit(), using future.task_run_id is still the practical way to identify it here.
Example for logging as tasks complete:
python
from prefect.client.orchestration import get_client
from prefect.futures import as_completed
from prefect.logging import get_run_logger
async def wait_for_tasks_with_logging(tasks):
logger = get_run_logger()
async with get_client() as client:
for future in as_completed(tasks):
task_run = await client.read_task_run(future.task_run_id)
try:
future.result()
<http://logger.info|logger.info>(f"Task completed: {task_run.name}")
except Exception:
logger.error(f"Task failed: {task_run.name}")
raise
A few notes:
- future.state.name is the state name (Completed, Failed), not the task name
- future.task_run_id is what lets you look up the task run metadata
- future.result() is still the right fail-fast call inside as_completed(...)
If you want to avoid the extra API call for every completed future, another good pattern is to track names yourself when submitting:
python
futures_to_names = {
my_task.submit(x): f"my_task({x})"
for x in items
}
for future in as_completed(list(futures_to_names)):
name = futures_to_names[future]
future.result()
<http://logger.info|logger.info>(f"Task completed: {name}")
That’s often cleaner if you already know the logical name you want to log.
So I’d recommend:
- best for exact Prefect run name: client.read_task_run(future.task_run_id).name
- best for efficiency/simplicity: keep your own future -> label mapping
Related API docs:
- Prefect futures
- Prefect client orchestration API
If you want, I can help you write a wait_for_tasks_fail_fast_with_logging() helper that combines:
- as_completed
- first-failure raise
- task-name logging
- cancellation of remaining child flow runsNicholas Pini
08/07/2026, 12:33 PMfuture.task_run_id is deprecatedMarvin
08/07/2026, 12:33 PMMarvin
08/07/2026, 12:34 PMtask_run_id is deprecated on the base PrefectFuture interface
- but there is not really a better direct replacement on the generic future for “give me the task name”
So if you want to avoid the deprecated property entirely, the best pattern is:
don’t discover the name from the future afterward — attach the name when you create the future.
Example:
python
from prefect.futures import as_completed
from prefect.logging import get_run_logger
async def wait_for_tasks_with_logging(tasks_by_name):
logger = get_run_logger()
for future in as_completed(list(tasks_by_name.keys())):
name = tasks_by_name[future]
try:
future.result()
<http://logger.info|logger.info>(f"Task completed: {name}")
except Exception:
logger.error(f"Task failed: {name}")
raise
Usage:
python
tasks_by_name = {}
for i in range(10):
future = my_task.submit(i)
tasks_by_name[future] = f"my_task({i})"
await wait_for_tasks_with_logging(tasks_by_name)
This is the cleanest solution because:
- no deprecated API
- no extra API lookup
- works with as_completed
- you control exactly what gets logged
If you truly need the actual Prefect task run name assigned by Prefect, then the deprecation makes things awkward, because there is no equally convenient replacement hanging off the generic future. In that case, you’d typically either:
1. keep your own mapping at submit time, or
2. use lower-level/client access and accept that you're stepping outside the nicest public abstraction
So my recommendation is:
use a dict[PrefectFuture, str] mapping from future to your desired display name.
Example with fail-fast:
python
from prefect.futures import as_completed
from prefect.logging import get_run_logger
async def wait_for_tasks_fail_fast(tasks_by_name):
logger = get_run_logger()
for future in as_completed(list(tasks_by_name.keys())):
name = tasks_by_name[future]
try:
future.result()
<http://logger.info|logger.info>(f"Task completed: {name}")
except Exception:
logger.exception(f"Task failed: {name}")
raise
If you want, I can show you a version that preserves your existing list[PrefectFuture] interface by taking a parallel list[str] or building the mapping automatically.Nicholas Pini
08/07/2026, 1:18 PMMarvin
08/07/2026, 1:18 PMMarvin
08/07/2026, 1:20 PMFlow object itself.
Use prefect.tags(...) when invoking the flow:
python
from prefect import flow, tags
@flow
def my_flow():
pass
with tags("prod", "important"):
my_flow()
That tags the resulting flow run.
A few distinctions:
- Flow object itself: no tags= parameter on @flow
- Flow run tags: use with tags(...)
- Task definition tags: supported on @task(tags={...})
- Deployment tags: set when creating/serving/deploying the deployment
Example with an existing flow object:
python
from prefect import tags
with tags("foo", "bar"):
flow_obj()
If you're asking about a deployment instead of a direct flow call, you can set tags there too, e.g. with serve() / deployment creation.
Docs:
- Tags and task runs
- `prefect.tags()` API
If you want, I can also show the difference between:
- tagging a direct flow call
- tagging a subflow call
- tagging a deployment runNicholas Pini
08/07/2026, 1:21 PMMarvin
08/07/2026, 1:21 PMMarvin
08/07/2026, 1:21 PMwith tags(...) propagate through the runtime context, so they apply to:
- the flow run you call inside the context
- subflows it calls
- subflows of those subflows
- tasks created within that context too
Example:
python
from prefect import flow, tags
from prefect.runtime import flow_run
@flow
def level_3():
print(flow_run.tags)
@flow
def level_2():
print(flow_run.tags)
level_3()
@flow
def level_1():
print(flow_run.tags)
level_2()
with tags("prod", "batch"):
level_1()
All three flow runs will inherit those tags.
Also, nested tag contexts are unioned, not replaced:
python
with tags("outer"):
with tags("inner"):
level_1()
Then descendants see both outer and inner.
So the short answer is: yes, tags propagate to subflows recursively through the call chain.
Docs:
- `prefect.tags()` API