Rajeshwar Agrawal
11/14/2022, 5:37 AMRunning
and does not changes state ever
2. If the work queue is started again, the flow-runs which were supposedly still running do not continue to run and are stuck in Running
state.
3. The work queue does not accept any other pending flow-runs after restarting, and its status remains Unhealthy
The expected behaviour is that once the agent is terminated, the associated Running
flow-runs are marked as Cancelled
or a similar state. Further, once the agent associated with work queue comes back up, it should be able to pickup new flow-runs submitted with its label
Steps to reproduce
1. Start a local prefect orion server
2. Create a work queue test
with parallelism 1
3. Build and deploy the attached flow log_flow.py
4. Submit 20 jobs at once repeat 20 prefect deployment run 'log-flow/log-flow'
5. Terminate the work queue process (close the terminate which started the work queue)
Observe that work queue status changes to unhealthy
however 1 flow runs that started running still show Running
state. Further on restarting the queue, the interrupted flow-run does not resume and no new pending flow runs are submitted to the work queuemerlin
11/14/2022, 5:17 PMMason Menges
11/14/2022, 6:23 PM