Hey @Prefect, We have started testing prefect orion (2.6.7) to start migrating from our existing self-host prefect community v1 edition I was testing an edge case, where prefect agent was terminated after submitting a bunch of flows to it. I observed that if the work queue is terminated with jobs already running on it -- 1. The flow-run status still showed

Running

and does not changes state ever 2. If the work queue is started again, the flow-runs which were supposedly still running do not continue to run and are stuck in

Running

state. 3. The work queue does not accept any other pending flow-runs after restarting, and its status remains

Unhealthy

The expected behaviour is that once the agent is terminated, the associated

Running

flow-runs are marked as

Cancelled

or a similar state. Further, once the agent associated with work queue comes back up, it should be able to pickup new flow-runs submitted with its label Steps to reproduce 1. Start a local prefect orion server 2. Create a work queue

test

with parallelism 1 3. Build and deploy the attached flow

log_flow.py

4. Submit 20 jobs at once

repeat 20 prefect deployment run 'log-flow/log-flow'

5. Terminate the work queue process (close the terminate which started the work queue) Observe that work queue status changes to

unhealthy

however 1 flow runs that started running still show

Running

state. Further on restarting the queue, the interrupted flow-run does not resume and no new pending flow runs are submitted to the work queue

I also have observed flows in Running state after a system restart. I'm only building a prototype in local environment, so I dont have all details. Deployed flow was running on agent queue, during long running task that connects to a DB service. Machine restarted, and after restoring the prefect agent and orion db the flow continued to show "Running" state for many days. I cant confirm if the agent picked up new scheduled flows or not.

Hey @Rajeshwar Agrawal @merlin we do have an open enhancement around this https://github.com/PrefectHQ/prefect/issues/7239