@Marvin sometimes tasks log that they have completed and downstream tasks (as defined by wait_for) are kicked off, but the prefect interface shows the task still in a running state (an hour or more later). Not sure if it's relevant but a later task did fail in at least one occurance of this, and although the flow oeverall is marked as failed that failed task (and some of the successfully completed tasks) are stuck in a running state. presumably this is just a bug that will be fixed somewhere by the prefect devs rather than something i've done wrong?

It seems you're not alone in facing this issue. There are several related bug reports in the Prefect GitHub repository about tasks or flows being stuck in a "Running" state even when they should have transitioned to another state. Here are some relevant issues: 1. Race Condition in Sub-deployments: There is a known issue where child deployments remain stuck in a "Canceling" state due to a race condition when a parent deployment is canceled. This is a non-deterministic issue and might not always occur. Issue #15399 2. Agent Termination: If an agent is stopped (e.g., by a

CTRL+C

), flows may remain in a "Running" state. Issue #7239 3. State Transition Errors: Sometimes, tasks show as "pending" or "running" due to a duplicate timestamp insert error in the

task_run_state

table, which violates unique constraints and can break the UI. Issue #16305 4. Timeout and Subflow Issues: Flows with subflows that crash and exceed a timeout can remain in a "Running" state without transitioning properly. Issue #7683 These issues suggest that the problem is more likely related to Prefect's handling of task and flow states rather than something specific you've done. You might want to keep an eye on the Prefect GitHub repository for updates or workarounds on these issues. If the problem persists or affects your workflows significantly, consider reaching out to the Prefect community or support for further assistance.