Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all, I believe there was a system outage today with Prefect Cloud.  We have a number of runs that were scheduled and are still stuck in a state of 'Cancelling'.  I believe my colleague already took care of re-running any necessary jobs earlier, shortly after the outage was resolved.

I am wondering how to kill the runs that are still flagged as 'Cancelling', as they keep triggering 'long running job' alerts in our main alerting system.  Thanks

IE there doesn't seem to be a way to cancel something which is already 'Cancelling' but that cancelling status has lasted for many hours now, when these jobs typically finish running in a few mins.

hi <@U077D3JA1C0>!

while its possible there was an interruption in service (though I'm not aware of one today), this can sometimes happen if your flow run's infra disappears (OOMs etc) while cancelling, such that the Prefect server-side never gets any state updates

you can resolve this a couple ways:
• (easiest imo) just delete the flow runs
• (more precisely addressing your request) you can `read_flow_runs` and <https://github.com/PrefectHQ/prefect/blob/6c45d953d6171b0c2d04d8ebcc00ea6ee87620ce/src/prefect/client/orchestration.py#L2152|then >`set_flow_run_state`<https://github.com/PrefectHQ/prefect/blob/6c45d953d6171b0c2d04d8ebcc00ea6ee87620ce/src/prefect/client/orchestration.py#L2152| with >`force=True` to Failed or Completed or Cancelled (both client methods) so itd be similar to the examples on <https://docs.prefect.io/3.0/manage/interact-with-api|this> page

Hey <@U02GMEZU18B>, thanks for the reply!  I think just deleting them sounds right, since another engineer already reran any jobs which needed.  How can I do this through the Prefect Cloud GUI?  I didn't see any way to cancel or delete a run that was already in a state of 'Cancelling'

any options through the UI are highly preferred in my case... I mainly work on the transformation jobs, and all the problematic jobs I mentioned are data ingestion pipelines.  It just happens to be my turn to be on-call for monitoring this stuff today :slightly_smiling_face:

image.png

totally understand!

if you go to

• Runs page
• Filter on states for Cancelling
• select the tick mark for each flow run to delete (or the select all box)


Thanks again, I'm starting to suspect it could be an access issue.  I see all the runs, but I cannot select or act on them.  At least not the ones which are currently marked 'Cancelling'

aha! your user may have a Runner or read-only role in that workspace which would prevent you from certain deletions / edits

makes sense now why I couldn't find the delete/cancel options.

thanks again for your help!  enjoy your weekend

no problem! you too :slightly_smiling_face: