I think I may have seen this asked somewhere before, but can no longer find the discussion, so apologies if this is duplicative, but.... I'm wondering if there's a way to know if a flow run running in a process (i.e. Process infra type) has been killed. I'm currently seeing these flow runs stay in status "running" when this happens, and im writing a cleanup that will reschedule these runs, but I'm having trouble identifying which flow runs have actually had their processes terminated

Hey Alix, what version of Prefect are you working with?

oh, sorry. 2.3.1

Hi Alix, a minimal reproducible example would be great. Could you share more about your setup? Generally speaking flow runs are not meant to be running as long running daemon processes or jobs, so it might be that some small workflow redesign would solve your issue. Prefect doesn't have access to your execution layer, but you can simply list Linux processes on that machine that have the work prefect in it and check their status this way

yeah, so we're running our agents in a heroku-like setup, and we have no control over the termination life cycle, so we basically have to assume that our agent containers can get sigkilled at any moment without any grace period. we're using github storage with local processes to run the flows. I have the main agent process doing cleanup of the flow runs its kicked off already, but because we have no control over the termination grace period, I'm going to need a "backup plan" so that another process could potentially clean up the runs.

So preemptive instances like spot instances on AWS? I'd recommend using this only for non-mission critical workflows since it's hard to make it work reliably as of today Prefect can orchestrate your dataflow but not infrastructure such as spot instances

I think i figured out a solution to my problem in using the datetime blocks to perform ping-like actions for flow runs. no, we weren't trying to manage our infra, we're just running in a very restrictive infrastructure (for now), so i've had to do some backflips to get things to look clean in the UI might post something in the discourse, super cool what you can do with blocks + blocks api for not much code

Wow, this sounds really cool! It would be great if you could share it in any way, Discourse or even a GitHub Gist

Hey @Alix Cook! Not sure exactly how you have things set up, but wanted to let you know that you could use

is_anonymous=True

when you create a DateTime Block so that you don’t need to create a named block. 🙂

we rely on the named blocks to be able to reference them in other flows. am i misunderstanding how anonymous blocks work?

Gotcha. Makes sense.