https://prefect.io logo
Title
r

Rajeshwar Agrawal

11/14/2022, 5:37 AM
Hey @Prefect, We have started testing prefect orion (2.6.7) to start migrating from our existing self-host prefect community v1 edition I was testing an edge case, where prefect agent was terminated after submitting a bunch of flows to it. I observed that if the work queue is terminated with jobs already running on it -- 1. The flow-run status still showed
Running
and does not changes state ever 2. If the work queue is started again, the flow-runs which were supposedly still running do not continue to run and are stuck in
Running
state. 3. The work queue does not accept any other pending flow-runs after restarting, and its status remains
Unhealthy
The expected behaviour is that once the agent is terminated, the associated
Running
flow-runs are marked as
Cancelled
or a similar state. Further, once the agent associated with work queue comes back up, it should be able to pickup new flow-runs submitted with its label Steps to reproduce 1. Start a local prefect orion server 2. Create a work queue
test
with parallelism 1 3. Build and deploy the attached flow
log_flow.py
4. Submit 20 jobs at once
repeat 20 prefect deployment run 'log-flow/log-flow'
5. Terminate the work queue process (close the terminate which started the work queue) Observe that work queue status changes to
unhealthy
however 1 flow runs that started running still show
Running
state. Further on restarting the queue, the interrupted flow-run does not resume and no new pending flow runs are submitted to the work queue
FYI @David Haines
m

merlin

11/14/2022, 5:17 PM
I also have observed flows in Running state after a system restart. I'm only building a prototype in local environment, so I dont have all details. Deployed flow was running on agent queue, during long running task that connects to a DB service. Machine restarted, and after restoring the prefect agent and orion db the flow continued to show "Running" state for many days. I cant confirm if the agent picked up new scheduled flows or not.
m

Mason Menges

11/14/2022, 6:23 PM
Hey @Rajeshwar Agrawal @merlin we do have an open enhancement around this https://github.com/PrefectHQ/prefect/issues/7239