<@ULVA73B9P> In Prefect 3, what happens if I call ...
# ask-marvin
n
@Marvin In Prefect 3, what happens if I call
wait()
on a list of tasks and one of them fails? Will the flow fail too?
m
thought for 60.3 seconds
In Prefect 3,
wait()
on a list of task futures does not itself raise if one task fails. What happens: -
wait(futures)
waits until the futures reach a terminal state - a failed task is still considered done - the flow does not automatically fail just because
wait()
saw a failed future The flow will typically fail only if you later do something like: -
future.result()
on the failed future, or - otherwise explicitly raise based on its state Example:
Copy code
python
from prefect import flow, task
from prefect.futures import wait

@task
def maybe_fail(x):
    if x == 2:
        raise ValueError("boom")
    return x * 10

@flow
def my_flow():
    futures = [maybe_fail.submit(i) for i in range(4)]

    done, not_done = wait(futures)

    print(f"done={len(done)}, not_done={len(not_done)}")

    for f in futures:
        print(f.state)

    # This is where failure would surface:
    return [f.result() for f in futures]
In that flow: -
wait(futures)
completes normally - one future will have a failed state - the flow fails when
[f.result() for f in futures]
hits the failed one If you want to handle partial failures gracefully, use one of these patterns:
Copy code
python
results = []
for f in futures:
    if f.state.is_failed():
        results.append(None)
    else:
        results.append(f.result())
or:
Copy code
python
results = [f.result(raise_on_failure=False) for f in futures]
With
raise_on_failure=False
, failed task results come back as exception objects instead of being raised. So the short answer is: - Will
wait()
raise?
No - Will the flow fail too? Not from
wait()
alone - When does it fail? Usually when you access the failed future’s result If you want, I can also show the exact behavior of
wait()
vs
as_completed()
vs
gather()
in Prefect 3.
n
@Marvin If I pass timeout=None to wait_for_flow_run, will it wait forever or will it not wait at all?
m
thought for 57.6 seconds
timeout=None
means wait indefinitely. It does not return immediately. So for `wait_for_flow_run(...)`: - default is
10800
seconds (3 hours) -
timeout=None
disables the timeout - it waits until the flow run reaches a final state Example:
Copy code
python
await wait_for_flow_run(flow_run_id, timeout=None)
That will block forever unless the flow run eventually finishes, fails, or is cancelled. One caveat: if the flow run gets stuck and never reaches a final state, your caller will also wait forever. If you want, I can also show the exact exception behavior for a numeric timeout.
n
@Marvin
Copy code
with LOCK:
        flow_run = await arun_deployment(
            name=flow_name,
            flow_run_name=flow_run_name,
            parameters=dict(**custom_args, s3_output_folder=s3_output_path.as_uri(), index_range=index_range),
            job_variables=jv,
        )
        append_uuid(flow_run.id)
    await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)

    assert flow_run.state is not None
    try:
        await flow_run.state.aresult()
    except Exception as e:
        logger.error("pet-wall-aspect-wall-height failed")
        raise e

    <http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
why did this code fail with
Copy code
Task run failed with exception: UnfinishedRun('Run is in SCHEDULED state, its result is not available.')
Traceback (most recent call last):
  File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1596, in run_context
    yield self
  File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1675, in run_task_async
    await engine.call_task_fn(txn)
  File "/opt/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 1613, in call_task_fn
    result = await call_with_parameters(self.task.fn, parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.12/site-packages/pet_wrapper/prepare_tiles.py", line 2571, in prepare_tiles
    raise e
  File "/opt/.venv/lib/python3.12/site-packages/pet_wrapper/prepare_tiles.py", line 2568, in prepare_tiles
    await flow_run.state.aresult()
  File "/opt/.venv/lib/python3.12/site-packages/prefect/client/schemas/objects.py", line 286, in aresult
    return await get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.12/site-packages/prefect/states.py", line 85, in get_state_result
    return await _get_state_result(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.12/site-packages/prefect/states.py", line 150, in _get_state_result
    raise UnfinishedRun(
prefect.exceptions.UnfinishedRun: Run is in SCHEDULED state, its result is not available.
it's like it has never waited for the run to finish
m
thought for 114.1 seconds
Yes — the problem is that
wait_for_flow_run()
waited, but you ignored the updated
FlowRun
object it returns
. Your code does this:
Copy code
python
flow_run = await arun_deployment(...)
await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
await flow_run.state.aresult()
But
flow_run
there is still the original object returned by
arun_deployment
, and that object can still have the old state like
SCHEDULED
.
wait_for_flow_run()
does not mutate that object in place. It returns a fresh
FlowRun
fetched from the API once the run reaches a final state. So this is the correct pattern:
Copy code
python
flow_run = await arun_deployment(
    name=flow_name,
    flow_run_name=flow_run_name,
    parameters=dict(
        **custom_args,
        s3_output_folder=s3_output_path.as_uri(),
        index_range=index_range,
    ),
    job_variables=jv,
)

append_uuid(flow_run.id)

flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)

assert flow_run.state is not None
try:
    await flow_run.state.aresult()
except Exception as e:
    logger.error("pet-wall-aspect-wall-height failed")
    raise

<http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
Why you saw
SCHEDULED
specifically
-
arun_deployment()
initially creates the run in a scheduled state - that returned Python object is just a snapshot - the actual run may later move to
RUNNING
,
COMPLETED
,
FAILED
, etc. - your original
flow_run
variable does not auto-refresh -
flow_run.state.aresult()
checks that stale state object, sees
SCHEDULED
, and raises
UnfinishedRun
So the failure message:
Copy code
python
UnfinishedRun('Run is in SCHEDULED state, its result is not available.')
does not mean
wait_for_flow_run()
failed to wait. It means you called
aresult()
on the stale pre-wait object. One more thing: if you're already using
wait_for_flow_run(..., timeout=None)
, you probably don't need both: -
arun_deployment(...)
- then
wait_for_flow_run(...)
Depending on your usage,
arun_deployment
may already support waiting behavior, but the safest fix for your current code is simply to reassign:
Copy code
python
flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
If you want, I can also help simplify this whole block into the cleanest Prefect 3 pattern for “run deployment and raise if child flow failed.”
n
@Marvin The problem is that the subflow never finished, so wait_for_flow_run DID NOT block the code
m
thought for 199.8 seconds
You're right — if the subflow truly never finished and your code continued past:
Copy code
python
await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
then
wait_for_flow_run()
did not behave like an infinite blocker in that execution. Based on the implementation, the most likely explanation is:
wait_for_flow_run()
relies on an event subscription, and that subscription can terminate early.
Most important case: - if the WebSocket/event stream closes cleanly - Prefect's event subscriber can stop iterating -
wait_for_flow_run()
can exit without the target run reaching a final state So in practice,
timeout=None
means “no timeout on the wait scope,” but not necessarily “guaranteed to block forever no matter what happens to the event stream.” That matches your observation: - child flow still unfinished - parent continued - stale
flow_run.state
still said
SCHEDULED
-
aresult()
then raised
UnfinishedRun
What I’d recommend Instead of trusting
wait_for_flow_run()
alone for this case, use a polling-based wait or hybrid wait. Example polling approach:
Copy code
python
import asyncio
from prefect.client.orchestration import get_client

async def wait_until_final(flow_run_id, poll_interval=5):
    async with get_client() as client:
        while True:
            flow_run = await client.read_flow_run(flow_run_id)
            if flow_run.state is not None and flow_run.state.is_final():
                return flow_run
            await asyncio.sleep(poll_interval)
Then:
Copy code
python
with LOCK:
    flow_run = await arun_deployment(
        name=flow_name,
        flow_run_name=flow_run_name,
        parameters=dict(
            **custom_args,
            s3_output_folder=s3_output_path.as_uri(),
            index_range=index_range,
        ),
        job_variables=jv,
    )
    append_uuid(flow_run.id)

flow_run = await wait_until_final(flow_run.id)

try:
    await flow_run.state.aresult()
except Exception:
    logger.error("pet-wall-aspect-wall-height failed")
    raise

<http://logger.info|logger.info>("pet-wall-aspect-wall-height finished successfully")
Why this is safer - it does not depend on the events/WebSocket channel staying alive - every loop iteration fetches fresh state from the API - if the run is still
SCHEDULED
or
RUNNING
, it keeps waiting Important nuance There are really two separate issues here: 1. you must reassign the returned
FlowRun
2. in your case,
wait_for_flow_run()
itself also seems to have returned prematurely Given your symptom, I think both are true. If you want, I can help you turn your exact block into a hardened helper like
run_deployment_and_wait()
with: - polling - optional timeout - child-failure propagation - better logging of stuck
SCHEDULED
runs
n
@Marvin what is the point of wait_for_flow_run if is so unreliable
m
thought for 7.0 seconds
Fair question. The point of
wait_for_flow_run()
is: - convenience - low-latency waiting - event-driven monitoring instead of polling the API repeatedly In the happy path, it’s nice because it: - subscribes to flow-run state change events - wakes up as soon as the run reaches a final state - avoids constant
read_flow_run(...)
calls So conceptually it exists to be a lightweight “wait until this flow run finishes” helper. But the tradeoff is that it depends on the event stream infrastructure working end-to-end: - client event subscriber - websocket connection - server event delivery - no disconnects or subscriber edge cases That makes it less robust than brute-force polling for critical orchestration. So I’d frame it like this: - Good for: normal interactive waiting, simple scripts, happy-path orchestration - Less ideal for: mission-critical parent/child orchestration where “must not proceed until final state” is a hard requirement For that stricter use case, polling is often the more reliable primitive:
Copy code
python
async def wait_until_final(flow_run_id, poll_interval=5):
    async with get_client() as client:
        while True:
            flow_run = await client.read_flow_run(flow_run_id)
            if flow_run.state and flow_run.state.is_final():
                return flow_run
            await asyncio.sleep(poll_interval)
So the point of
wait_for_flow_run()
is not “guaranteed strongest synchronization primitive,” but “efficient event-based waiting helper.” Given your use case, I would not rely on it alone. I’d use: - polling only, or - event wait with polling fallback If you want, I can give you a production-safe
wait_for_flow_run_strict()
helper you can drop into your codebase.
n
@Marvin why didn't wait for flow run work for me?
m
thought for 8.9 seconds
Most likely:
wait_for_flow_run()
was waiting on the event stream, and that stream ended before your child flow reached a final state. So the function felt like “wait until done,” but internally it is closer to: - subscribe to flow-run events - return when a final-state event is seen - but if the subscription stops unexpectedly, it may stop waiting For your case, the strongest explanation is: 1.
arun_deployment(...)
created the child run 2. child run stayed non-final (
SCHEDULED
/ not finished) 3.
wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
relied on Prefect events 4. that event subscription likely disconnected or terminated early 5. your code resumed 6. you then checked the old
flow_run.state
, which was still
SCHEDULED
7.
aresult()
raised
UnfinishedRun
So it didn't work for you because the waiting mechanism is event-driven, not a strict polling loop, and event delivery/connection appears to have failed in your run. A few plausible concrete causes: - websocket/event subscriber closed cleanly - network interruption between your runner and Prefect API - server-side event stream interruption - a bug/edge case in the subscriber loop - the child run never emitted the expected final-state event to that subscriber session What to do next For your use case, I’d treat
wait_for_flow_run()
as best-effort, not strict. Use polling instead:
Copy code
python
import asyncio
from prefect.client.orchestration import get_client

async def wait_until_final(flow_run_id, poll_interval=5):
    async with get_client() as client:
        while True:
            flow_run = await client.read_flow_run(flow_run_id)
            state = flow_run.state
            if state is not None and state.is_final():
                return flow_run
            await asyncio.sleep(poll_interval)
Then:
Copy code
python
flow_run = await arun_deployment(...)
flow_run = await wait_until_final(flow_run.id)
await flow_run.state.aresult()
If you want to confirm this was an event-stream problem, add debug logging around the events client and compare timestamps:
Copy code
python
import logging
logging.getLogger("prefect.events.clients").setLevel(logging.DEBUG)
So the short answer is: it likely didn’t work because the event subscription ended before the child run finished, and
wait_for_flow_run()
is built on that subscription rather than guaranteed polling.
If you want, I can help you build a drop-in replacement with: - strict polling - timeout support - logging of state transitions - detection of “stuck in SCHEDULED” runs
n
@Marvin I don't understand why the event subscription failed, I don't have any issues on my server
m
thought for 9.5 seconds
You may not have any obvious server issue and still hit this.
wait_for_flow_run()
depends on a live client-side event stream, so failure does not require “the server is broken.” It can happen even when the server is generally healthy. Possible reasons: - the websocket closed normally and the client treated that as end-of-stream - a transient network hiccup between your runner and Prefect API - a proxy / load balancer / ingress idle timeout - container/job lifecycle behavior in your execution environment - an edge case in Prefect’s event subscriber handling - the subscriber missed the event and had no polling fallback So “server looks fine” does not rule this out. In your environment, I’d especially suspect infrastructure between the runner and API: - AWS CodeBuild / CI jobs - NAT/proxy behavior - load balancer websocket handling - idle connection timeouts Because event-based waiting is more sensitive to those than normal API polling. Why this is confusing Most Prefect operations are request/response HTTP: - create run - read run - update state Those can all work perfectly. But
wait_for_flow_run()
is different: - it opens a longer-lived subscription - that is inherently more fragile than short API calls So you can have: - healthy Prefect server - healthy database - normal UI/API behavior - and still a broken or interrupted event wait in one client session How to verify 1. Turn on debug logs for the events client:
Copy code
python
import logging
logging.getLogger("prefect.events.clients").setLevel(logging.DEBUG)
2. Log around the wait:
Copy code
python
<http://logger.info|logger.info>(f"waiting for child flow run {flow_run.id}")
updated_flow_run = await wait_for_flow_run(flow_run.id, timeout=None, log_states=True)
<http://logger.info|logger.info>(f"wait returned with state={updated_flow_run.state}")
3. If possible, immediately compare with a fresh API read:
Copy code
python
async with get_client() as client:
    fresh = await client.read_flow_run(flow_run.id)
<http://logger.info|logger.info>(f"fresh state after wait: {fresh.state}")
If
wait_for_flow_run()
returns but a fresh read shows non-final state, that strongly confirms the wait mechanism exited early. Bottom line You do not need a broken server for this to happen. A healthy server plus: - websocket close - networking middlebox timeout - subscriber bug is enough. For anything critical, I’d recommend polling as the source of truth. If you want, I can give you a small diagnostic wrapper that logs: - event wait start/end - returned state - immediate fresh-read state - elapsed time so you can prove exactly where it breaks.
n
@Marvin Here is what I am doing. I have a bunch of tasks that call run_deployment with timeout=0, add the UUID to a global list, and wait for the run to finish. On the parent flow, I have a hook which is called on_failed, which cancels all runs whose UUID is in the global list. Finally, I wait for these tasks in the main flow like this
Copy code
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.state.aresult()
m
thought for 20.6 seconds
Thanks — that architecture explains a lot. The key issue is probably not just
wait_for_flow_run()
by itself, but the interaction between: -
run_deployment(..., timeout=0)
- child runs tracked in a global list - parent
on_failure
hook canceling those child runs - tasks awaiting child completion -
as_completed(tasks)
in the parent flow What your pattern is effectively doing 1. A task starts a child deployment with
timeout=0
- so it returns immediately - child flow run may still be
SCHEDULED
or
RUNNING
2. The task stores the child run id globally 3. The task then waits for that child run to finish 4. If one sibling task fails, the parent flow can fail 5. Your parent
on_failure
hook runs and cancels all child runs in the global list 6. Other tasks that were waiting on child runs may now see cancellation / non-final / stale state behavior That means you have real concurrency between: - child runs progressing - parent flow failure - cancellation hook firing - task futures completing/failing - your
wait_for_tasks()
loop consuming task results That can produce confusing states. The most important problem in your code is here:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.state.aresult()
I would not rely on
task.state.aresult()
on the future's
state
attribute like that. Why: -
task.state
may be stale -
as_completed(tasks)
tells you the future completed, but the
state
attribute you read may not be the freshest representation you expect - the normal future interface is
await task.result()
or
task.result()
, not
task.state.aresult()
You should prefer:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.result()
or if async result access isn't available on that future type in your context, then at least:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        state = await task.wait()
        await task.result()
But the cleanest is usually just:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.result()
Why this matters
PrefectFuture.state
is not the same thing as “always current authoritative task state from the API.” When you do:
Copy code
python
await task.state.aresult()
you're calling
aresult()
on whatever
State
object is attached there at that moment. If that state is stale or non-final, you can get misleading exceptions. By contrast,
task.result()
is the future abstraction meant to surface completion/failure properly. Another likely issue: parent failure causes child cancellation while siblings are still waiting Suppose: - one task fails - parent flow begins failing -
on_failure
hook cancels child runs - another task is still in
wait_for_flow_run(child_id, timeout=None)
Now that waiting task may: - return due to event-stream weirdness - see child cancellation - see stale flow-run state - fail while trying to read result So your observed behavior may be a consequence of cancellation orchestration, not just a standalone wait bug. Global list is also risky A global mutable list of run IDs can be problematic in flows/tasks because: - concurrent tasks mutate shared state - retries can duplicate IDs - task runner/process boundaries may make globals unreliable - failure hooks may run with partial or stale contents Safer patterns are: - return child run IDs from tasks and collect them in the flow - or persist them in Prefect artifacts / external storage / a thread-safe structure - or model orchestration so the flow owns child-run bookkeeping explicitly What I would change first 1. Replace this:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.state.aresult()
with this: ```python
async def wait_for_tasks(tasks: list[PrefectFuture]): for task in as_completed(tasks): await task.result()
Copy code
2. Do not depend on `wait_for_flow_run()` as the only child-wait primitive for critical logic; use polling.

3. Avoid reading `.state.aresult()` on stale flow-run objects; refresh first or use the future/result abstraction.

4. Reconsider global bookkeeping for child run IDs.

A more robust task pattern would look like:

```python
import asyncio
from prefect import task
from prefect.client.orchestration import get_client
from prefect.deployments.flow_runs import run_deployment

async def wait_until_final(flow_run_id, poll_interval=5):
    async with get_client() as client:
        while True:
            flow_run = await client.read_flow_run(flow_run_id)
            if flow_run.state and flow_run.state.is_final():
                return flow_run
            await asyncio.sleep(poll_interval)

@task
async def launch_and_wait(...):
    flow_run = await run_deployment(
        name="my-deployment",
        parameters={...},
        timeout=0,
    )

    append_uuid(flow_run.id)

    flow_run = await wait_until_final(flow_run.id)

    await flow_run.state.aresult()
    return flow_run.id
Then:
Copy code
python
async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.result()
Bottom line I think your issue is likely a combination of: - using
wait_for_flow_run()
in a critical synchronization path - then reading stale
.state
- plus parent failure/cancellation racing with child waits - plus using
task.state.aresult()
instead of the future’s
result()
interface If you want, I can help you redesign this into a safer parent/child orchestration pattern that preserves: - “launch many child deployments” - “cancel all children if parent fails” - “surface the first child failure cleanly” without relying on globals and fragile event waits.
n
@Marvin I was able to make it work by using wait_for_flow_run in the tasks (so this was NOT an issue) and by using the wait function from prefect.futures. That said, how can I wait for a list of tasks, but throw an error AS SOON AS one task fails? I don't want to wait for all tasks to finish before checking which one has failed, if any
m
thought for 83.8 seconds
Yes — use
prefect.futures.as_completed()
and call
.result()
on each future as it completes. That gives you fail-fast behavior: as soon as the first completed task is in a failed state,
.result()
raises immediately, and you do not wait for the rest. Pattern:
Copy code
python
from prefect.futures import as_completed

async def wait_for_tasks_fail_fast(tasks):
    for task in as_completed(tasks):
        await task.result()
If your futures are not awaitable with
await task.result()
in your context, use:
Copy code
python
from prefect.futures import as_completed

def wait_for_tasks_fail_fast(tasks):
    for task in as_completed(tasks):
        task.result()
The important part is: -
as_completed(tasks)
yields futures in completion order -
task.result()
raises on failure by default - so the first failed task to finish stops the loop immediately This is exactly what you want, and it’s different from: -
wait(tasks)
→ waits for all tasks to finish - iterating the original list and calling
.result()
→ can block on an earlier slow task and miss a later fast failure So your fail-fast version should be:
Copy code
python
from prefect.futures import as_completed, PrefectFuture

async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        await task.result()
If you also want to cancel sibling child flow runs as soon as one fails, I can show how to combine this with your global UUID cancellation logic so you fail fast and actively stop the remaining work.
n
@Marvin I already tried that fix, and I got an error along the lines of "cannot await on None" where I called "await task.result()"
m
thought for 9.7 seconds
That makes sense — for Prefect futures,
.result()
is usually not an async coroutine, so
await task.result()
is wrong. Use:
Copy code
python
from prefect.futures import as_completed, PrefectFuture

async def wait_for_tasks(tasks: list[PrefectFuture]):
    for task in as_completed(tasks):
        task.result()
Not:
Copy code
python
await task.result()
Why this works: -
as_completed(tasks)
only yields futures once they are done -
task.result()
then immediately returns the result or raises the task exception - if one task failed,
task.result()
raises right there - so this is still fail-fast So the correct fail-fast pattern is:
Copy code
python
from prefect.futures import as_completed

async def wait_for_tasks_fail_fast(tasks):
    for task in as_completed(tasks):
        task.result()
If you want to preserve the async function signature, that’s fine — just don’t
await
the
.result()
call. If you'd like, I can also show you how to cancel the remaining tasks / child deployments immediately when one of those
task.result()
calls raises.
n
@Marvin show me
m
thought for 8.9 seconds
Sure — wrap the
as_completed()
loop in
try/except
, and when one task fails, cancel the remaining child flow runs before re-raising. Something like this:
Copy code
python
from prefect.futures import as_completed, PrefectFuture
from prefect.client.orchestration import get_client

# however you store these
CHILD_FLOW_RUN_IDS: set[str] = set()

async def cancel_child_flow_runs(flow_run_ids):
    async with get_client() as client:
        for flow_run_id in flow_run_ids:
            try:
                await client.cancel_flow_run(flow_run_id)
            except Exception:
                # optional: log and continue
                pass

async def wait_for_tasks_fail_fast(tasks: list[PrefectFuture]):
    try:
        for task in as_completed(tasks):
            task.result()  # raises immediately if this task failed
    except Exception:
        await cancel_child_flow_runs(CHILD_FLOW_RUN_IDS)
        raise
If your IDs are UUIDs instead of strings, that’s fine too. A slightly safer version tracks only the runs that are still active:
Copy code
python
from prefect.futures import as_completed, PrefectFuture
from prefect.client.orchestration import get_client

CHILD_FLOW_RUN_IDS: set = set()

async def cancel_child_flow_runs(flow_run_ids):
    async with get_client() as client:
        for flow_run_id in flow_run_ids:
            try:
                await client.cancel_flow_run(flow_run_id)
            except Exception:
                pass

async def wait_for_tasks_fail_fast(tasks: list[PrefectFuture]):
    try:
        for task in as_completed(tasks):
            task.result()
    except Exception:
        await cancel_child_flow_runs(CHILD_FLOW_RUN_IDS)
        raise
And if you want logging:
Copy code
python
from prefect.futures import as_completed
from prefect.client.orchestration import get_client

async def cancel_child_flow_runs(flow_run_ids, logger):
    async with get_client() as client:
        for flow_run_id in flow_run_ids:
            try:
                <http://logger.info|logger.info>(f"Cancelling child flow run {flow_run_id}")
                await client.cancel_flow_run(flow_run_id)
            except Exception as exc:
                logger.warning(f"Failed to cancel child flow run {flow_run_id}: {exc}")

async def wait_for_tasks_fail_fast(tasks, child_flow_run_ids, logger):
    try:
        for task in as_completed(tasks):
            task.result()
    except Exception as exc:
        logger.error(f"A task failed, cancelling remaining child flow runs: {exc}")
        await cancel_child_flow_runs(child_flow_run_ids, logger)
        raise
A couple of notes: -
task.result()
is the thing that raises the task failure -
as_completed()
gives you fail-fast behavior based on completion order - canceling child flow runs won’t automatically cancel already-finished ones, so it’s okay if some IDs are stale - if multiple tasks can mutate the global set concurrently, use a lock or a copy before iterating For example:
Copy code
python
await cancel_child_flow_runs(set(CHILD_FLOW_RUN_IDS), logger)
If you want, I can help you make this fully race-safe with: - a lock around the run-id set - removing IDs after successful completion - avoiding duplicate cancel attempts
n
@Marvin Is
client.cancel_flow_run
available in prefect 3.6.12?
m
thought for 90.6 seconds
No —
client.cancel_flow_run
is not available in Prefect 3.6.12. The supported Python-client way is to set the flow run state to
Cancelling
via
set_flow_run_state(...)
. Example:
Copy code
python
from prefect.client.orchestration import get_client
from prefect.states import Cancelling

async def cancel_flow_run(flow_run_id):
    async with get_client() as client:
        await client.set_flow_run_state(
            flow_run_id,
            Cancelling(message="Cancelled because a sibling task failed"),
        )
So if you want to cancel remaining child flow runs on first failure, use that pattern instead of
client.cancel_flow_run(...)
. If helpful, your earlier helper should become:
Copy code
python
from prefect.client.orchestration import get_client
from prefect.states import Cancelling

async def cancel_child_flow_runs(flow_run_ids):
    async with get_client() as client:
        for flow_run_id in flow_run_ids:
            try:
                await client.set_flow_run_state(
                    flow_run_id,
                    Cancelling(message="Cancelled because a sibling task failed"),
                )
            except Exception:
                pass
Related docs: - Cancel workflows - Python client API One important note: setting
Cancelling
requests cancellation; actual stopping depends on the worker/infrastructure picking that up.
n
@Marvin When I use
.submit(wait_for=list_of_tasks)
, I get this LSP error:
Copy code
1. No overloads for "submit" match the provided arguments
     Argument types: (str, list[PrefectFuture[CoroutineType[Any, Any, None]]]) [reportCallIssue]
list_of_tasks is a list of
PrefectFuture[CoroutineType[Any, Any, None]]
, and when I run the code it works correctly. Why do I get the LSP error?
m
thought for 186.0 seconds
This is a typing bug / overload mismatch, not a runtime problem. What’s happening: - at runtime,
.submit(wait_for=...)
accepts your list and works - but the type checker sees your async task futures as
PrefectFuture[CoroutineType[Any, Any, None]]
- the
submit()
overloads don’t fully account for that case unless async-task typing is narrowed correctly - so Pyright/Pylance reports “no overload matches,” even though Prefect executes it fine In short: the LSP is complaining about Prefect’s type hints, not your actual code behavior. Why specifically this happens with async tasks: - for an
async def
task, the raw Python function returns a coroutine - Prefect’s runtime unwraps that correctly - but the static typing around
Task.submit(...)
can infer the future as wrapping
Coroutine[...]
instead of the resolved result type - then `wait_for`’s overload no longer matches cleanly So your
list[PrefectFuture[CoroutineType[Any, Any, None]]]
is the clue: that type is what the LSP doesn’t like. Workarounds 1. Best workaround: pass
return_state=False
explicitly
Copy code
python
next_task.submit("arg", wait_for=list_of_tasks, return_state=False)
That can help the type checker choose the intended overload. 2. Cast the list if you want to silence the warning cleanly
Copy code
python
from typing import cast
from prefect.futures import PrefectFuture

deps = cast(list[PrefectFuture[None]], list_of_tasks)
next_task.submit("arg", wait_for=deps)
3. Use a targeted ignore if needed
Copy code
python
next_task.submit("arg", wait_for=list_of_tasks)  # type: ignore[arg-type]
My recommendation: - if runtime behavior is correct, treat this as a Prefect typing issue - use either explicit
return_state=False
or a
cast(...)
So the short answer is: you get the LSP error because Prefect’s
submit()
overload typing for async-task futures is imperfect, and your editor is enforcing the static types more strictly than runtime does.
If you want, I can help you write the least ugly cast/helper so this disappears throughout your codebase.
n
@Marvin How do I get a task name from a
PrefectFuture[CoroutineType[Any, Any, None]]
object? It has a
task_run_id
property but it is deprecated (I am using Prefect 3.6.12). I want to log a message each time a task in a list of tasks is completed
m
thought for 122.3 seconds
The short answer: there is not a direct public
future.task_name
property on a
PrefectFuture
. If you want the task run name, the supported way is: - get the future’s
task_run_id
- read the task run from the client - use
task_run.name
And importantly: if your future is a task future from
.submit()
, using
future.task_run_id
is still the practical way to identify it here. Example for logging as tasks complete:
Copy code
python
from prefect.client.orchestration import get_client
from prefect.futures import as_completed
from prefect.logging import get_run_logger

async def wait_for_tasks_with_logging(tasks):
    logger = get_run_logger()

    async with get_client() as client:
        for future in as_completed(tasks):
            task_run = await client.read_task_run(future.task_run_id)

            try:
                future.result()
                <http://logger.info|logger.info>(f"Task completed: {task_run.name}")
            except Exception:
                logger.error(f"Task failed: {task_run.name}")
                raise
A few notes: -
future.state.name
is the state name (
Completed
,
Failed
), not the task name -
future.task_run_id
is what lets you look up the task run metadata -
future.result()
is still the right fail-fast call inside
as_completed(...)
If you want to avoid the extra API call for every completed future, another good pattern is to track names yourself when submitting:
Copy code
python
futures_to_names = {
    my_task.submit(x): f"my_task({x})"
    for x in items
}

for future in as_completed(list(futures_to_names)):
    name = futures_to_names[future]
    future.result()
    <http://logger.info|logger.info>(f"Task completed: {name}")
That’s often cleaner if you already know the logical name you want to log. So I’d recommend: - best for exact Prefect run name:
client.read_task_run(future.task_run_id).name
- best for efficiency/simplicity: keep your own
future -> label
mapping Related API docs: - Prefect futures - Prefect client orchestration API If you want, I can help you write a
wait_for_tasks_fail_fast_with_logging()
helper that combines: -
as_completed
- first-failure raise - task-name logging - cancellation of remaining child flow runs
n
@Marvin
future.task_run_id
is deprecated
m
thought for 8.5 seconds
You're right to call that out. The nuance is: -
task_run_id
is deprecated on the base
PrefectFuture
interface - but there is not really a better direct replacement on the generic future for “give me the task name” So if you want to avoid the deprecated property entirely, the best pattern is: don’t discover the name from the future afterward — attach the name when you create the future. Example:
Copy code
python
from prefect.futures import as_completed
from prefect.logging import get_run_logger

async def wait_for_tasks_with_logging(tasks_by_name):
    logger = get_run_logger()

    for future in as_completed(list(tasks_by_name.keys())):
        name = tasks_by_name[future]
        try:
            future.result()
            <http://logger.info|logger.info>(f"Task completed: {name}")
        except Exception:
            logger.error(f"Task failed: {name}")
            raise
Usage:
Copy code
python
tasks_by_name = {}

for i in range(10):
    future = my_task.submit(i)
    tasks_by_name[future] = f"my_task({i})"

await wait_for_tasks_with_logging(tasks_by_name)
This is the cleanest solution because: - no deprecated API - no extra API lookup - works with
as_completed
- you control exactly what gets logged If you truly need the actual Prefect task run name assigned by Prefect, then the deprecation makes things awkward, because there is no equally convenient replacement hanging off the generic future. In that case, you’d typically either: 1. keep your own mapping at submit time, or 2. use lower-level/client access and accept that you're stepping outside the nicest public abstraction So my recommendation is: use a
dict[PrefectFuture, str]
mapping from future to your desired display name.
Example with fail-fast:
Copy code
python
from prefect.futures import as_completed
from prefect.logging import get_run_logger

async def wait_for_tasks_fail_fast(tasks_by_name):
    logger = get_run_logger()

    for future in as_completed(list(tasks_by_name.keys())):
        name = tasks_by_name[future]
        try:
            future.result()
            <http://logger.info|logger.info>(f"Task completed: {name}")
        except Exception:
            logger.exception(f"Task failed: {name}")
            raise
If you want, I can show you a version that preserves your existing
list[PrefectFuture]
interface by taking a parallel
list[str]
or building the mapping automatically.
n
@Marvin Given a flow object, how do I add tags to it before running it?
m
thought for 140.3 seconds
In Prefect 3, you generally add tags to the run context, not to the
Flow
object itself. Use
prefect.tags(...)
when invoking the flow:
Copy code
python
from prefect import flow, tags

@flow
def my_flow():
    pass

with tags("prod", "important"):
    my_flow()
That tags the resulting flow run. A few distinctions: - Flow object itself: no
tags=
parameter on
@flow
- Flow run tags: use
with tags(...)
- Task definition tags: supported on
@task(tags={...})
- Deployment tags: set when creating/serving/deploying the deployment Example with an existing flow object:
Copy code
python
from prefect import tags

with tags("foo", "bar"):
    flow_obj()
If you're asking about a deployment instead of a direct flow call, you can set tags there too, e.g. with
serve()
/ deployment creation. Docs: - Tags and task runs - `prefect.tags()` API If you want, I can also show the difference between: - tagging a direct flow call - tagging a subflow call - tagging a deployment run
n
@Marvin Does this also attach tags to subflows of subflows?
m
thought for 6.8 seconds
Yes — tags set with
with tags(...)
propagate through the runtime context, so they apply to: - the flow run you call inside the context - subflows it calls - subflows of those subflows - tasks created within that context too Example:
Copy code
python
from prefect import flow, tags
from prefect.runtime import flow_run

@flow
def level_3():
    print(flow_run.tags)

@flow
def level_2():
    print(flow_run.tags)
    level_3()

@flow
def level_1():
    print(flow_run.tags)
    level_2()

with tags("prod", "batch"):
    level_1()
All three flow runs will inherit those tags. Also, nested tag contexts are unioned, not replaced:
Copy code
python
with tags("outer"):
    with tags("inner"):
        level_1()
Then descendants see both
outer
and
inner
. So the short answer is: yes, tags propagate to subflows recursively through the call chain. Docs: - `prefect.tags()` API