<@ULVA73B9P>: How are the `not_done` from `wait` t...
# marvin-ai
j
@Marvin: How are the
not_done
from
wait
to be meaningfully used? For example, I have:
Copy code
# Wait for all chain tasks to complete
    result: DoneAndNotDoneFutures[ReportRow] = wait(chain_futures)
    done, not_done = result

    # Check each future's state
    results: list[ReportRow] = []
    failed: list[State] = []

    for future in done:
        if future.state.is_completed():
            results.append(future.result())
        else:
            failed.append(future.state)
Can I extract helpful information from the
not_done
for reporting?
m
thought for 268.1 seconds
Short answer: yes. The
not_done
set is most useful when you pass a timeout to
wait(...)
. Without a timeout,
wait
blocks until everything finishes, so
not_done
will be empty. What you can report from each
not_done
future: - Identity:
future.task_run_id
(or
future.flow_run_id
for subflows) - Current state:
future.state
with checks like
is_running()
,
is_pending()
,
is_scheduled()
- Timestamps and messages:
future.state.timestamp
,
future.state.updated
,
future.state.message
- Scheduling/meta:
future.state.state_details.scheduled_time
, plus IDs in
state_details
(e.g.,
task_run_id
,
flow_run_id
,
child_flow_run_id
) Example pattern that adds reporting for `not_done`:
Copy code
from prefect.futures import wait

# after submitting your tasks...
done, not_done = wait(chain_futures, timeout=30)  # use a timeout for partial results

results: list[ReportRow] = []
failed: list[State] = []
in_progress_report: list[dict] = []

# Handle completed/failed
for f in done:
    if f.state.is_completed():
        results.append(f.result())
    else:
        failed.append(f.state)

# Report on what’s still not done
for f in not_done:
    s = f.state
    in_progress_report.append(
        {
            "task_run_id": str(getattr(f, "task_run_id", None)),
            "state": s.name,                      # e.g. "Running", "Pending", "Scheduled"
            "is_running": s.is_running(),
            "is_pending": s.is_pending(),
            "is_scheduled": s.is_scheduled(),
            "message": s.message,
            "scheduled_for": getattr(s.state_details, "scheduled_time", None),
            "last_updated": getattr(s, "updated", None),
            "created": getattr(s, "timestamp", None),
        }
    )

# Optional: continue waiting later just for the remaining
if not_done:
    more_done, still_not_done = wait(list(not_done), timeout=60)
Tips - If you need to know “which input” a future corresponds to, keep a side-car mapping when you submit (e.g., a dict from future to the item/kwargs you passed) and include that in your report. - Futures in
not_done
continue running; you can call
future.wait()
or
future.result(timeout=...)
later if you want to block on them individually. Note that
future.result(timeout=...)
will raise
TimeoutError
if it still hasn’t finished, while
prefect.futures.wait(...)
never raises on timeout and instead returns the
not_done
set. Docs and source - API reference for
wait
and `DoneAndNotDoneFutures`: prefect.futures API - Source for `wait`: futures.py#L592 - State object reference (methods like
is_running/is_pending/...
and fields like
message
, timestamps, and
state_details
): State API
j
@Marvin:
- If you need to know “which input” a future corresponds to, keep a side-car mapping when you submit (e.g., a dict from future to the item/kwargs you passed) and include that in your report.
If I call
.map
on an
@task
, then it’s submitted and returned in order in my code snippet?
m
thought for 102.0 seconds
Yes. In Prefect 3.x,
Task.map(...)
preserves the order of the input iterable(s) and returns a
PrefectFutureList
whose items correspond 1:1 by index with the inputs. Implications: - You can safely index-match inputs to futures (e.g.,
zip(items, futures)
), or keep a side-car list/dict keyed by the position. -
futures.result()
also returns results in the same order as the mapped inputs. - This holds for normal and deferred execution paths. Docs/source for reference: - API: prefect.futures API (see
PrefectFutureList.result
order guarantee) - Source (map and ordering): - Task.map core flow: tasks.py#L1456-L1636 - TaskRunner mapping loop (index-based, preserves order): task_runners.py#L112-L219 - PrefectFutureList.result preserves order: futures.py#L487-L538
j
@Marvin: Okay so if I set
raise_on_failure=False
, I’ll get
Exception
s mixed into my results?
m
thought for 85.1 seconds
Short version: yes. Details: - For each future,
future.result(raise_on_failure=False)
returns: - The actual value if the state is completed - The exception object if the state is failed/crashed/cancelled - For a mapped set,
futures.result(raise_on_failure=False)
returns a list in input order, mixing real results and exception objects for failures. This is by design so you can post-process without raising:
Copy code
results = futures.result(raise_on_failure=False)
successes = []
failures = []
for r in results:
    if isinstance(r, Exception):
        failures.append(r)
    else:
        successes.append(r)
Refs: - Source (result ordering and behavior): futures.py#L487-L538 - State result semantics (returns exception when not raising): states.py#L134-L180