Ruslan
02/02/2022, 6:34 AMAnna Geller
02/02/2022, 11:38 AMRuslan
02/02/2022, 11:42 AMAnna Geller
02/02/2022, 1:00 PMRuslan
02/03/2022, 11:10 AM• if you want to get notified on a flow run cancellation, reacting to the Cancelled state still solves the problem - you will get the message informing you that the flow run has been cancelledI guess you didn’t understand me. One more time my flow freezes (I don’t know why, because of prefect some bugs). Automation cancel this flow after some time. But this flow went in status “cancelling” and NEVER completely cancels. Prefect INFINITELY try to cancel it but no luck. Your example catches prefect.engine.state.Cancelled yes it is work in your case, because your simple flow cancels, it not freezes. And this handler work when flow completely canceled, same way as working notifications from UI. It is not about my case, my flow does not reach the status “cancelled” and doesn’t catch this handler
Anna Geller
02/03/2022, 11:23 AMRuslan
02/03/2022, 2:58 PMAnna Geller
02/03/2022, 3:47 PMfrom prefect.run_configs import UniversalRun
flow.run_config = UniversalRun(env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})
Since you say that you struggle with this issue, I can try to reproduce the issue in the next days (rather next week) if you can give me as much information as you can 🙂Ruslan
02/03/2022, 5:03 PMAnna Geller
02/03/2022, 5:19 PMPREFECT__CLOUD__HEARTBEAT_MODE
to the run configuration. Since you are using a local agent, you can use either LocalRun or UniversalRun and attach it to your flow:
from prefect.run_configs import UniversalRun
with Flow("name", run_config = UniversalRun(env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})) as flow:
...
This helps a lot when you have issues as those that you see because the flow heartbeat’s mode then switches to threads rather than processes, which is more stable when you use many external services or long-running jobs.
Again, I totally understand the frustration especially if you don’t understand why this is happening. I wonder whether you could perhaps try to run a similar flow with Orion and see how this goes? Some features such as concurrency limits are not yet there, but maybe Orion can handle such flows better - would be great to test.Ruslan
02/03/2022, 6:38 PMAnna Geller
02/03/2022, 8:30 PM