https://prefect.io logo
Title
a

Alix Cook

09/02/2022, 10:23 PM
I think I may have seen this asked somewhere before, but can no longer find the discussion, so apologies if this is duplicative, but.... I'm wondering if there's a way to know if a flow run running in a process (i.e. Process infra type) has been killed. I'm currently seeing these flow runs stay in status "running" when this happens, and im writing a cleanup that will reschedule these runs, but I'm having trouble identifying which flow runs have actually had their processes terminated
1
m

Matt Conger

09/02/2022, 10:51 PM
Hey Alix, what version of Prefect are you working with?
a

Alix Cook

09/02/2022, 11:00 PM
oh, sorry. 2.3.1
I'm currently utilizing the DateTime block to basically do a ping every once in a while for a flow run, but was hoping for a smarter solution heh
a

Anna Geller

09/03/2022, 12:50 PM
Hi Alix, a minimal reproducible example would be great. Could you share more about your setup? Generally speaking flow runs are not meant to be running as long running daemon processes or jobs, so it might be that some small workflow redesign would solve your issue. Prefect doesn't have access to your execution layer, but you can simply list Linux processes on that machine that have the work prefect in it and check their status this way
a

Alix Cook

09/03/2022, 3:25 PM
yeah, so we're running our agents in a heroku-like setup, and we have no control over the termination life cycle, so we basically have to assume that our agent containers can get sigkilled at any moment without any grace period. we're using github storage with local processes to run the flows. I have the main agent process doing cleanup of the flow runs its kicked off already, but because we have no control over the termination grace period, I'm going to need a "backup plan" so that another process could potentially clean up the runs.
a

Anna Geller

09/03/2022, 5:38 PM
So preemptive instances like spot instances on AWS? I'd recommend using this only for non-mission critical workflows since it's hard to make it work reliably as of today Prefect can orchestrate your dataflow but not infrastructure such as spot instances
a

Alix Cook

09/07/2022, 2:44 AM
I think i figured out a solution to my problem in using the datetime blocks to perform ping-like actions for flow runs. no, we weren't trying to manage our infra, we're just running in a very restrictive infrastructure (for now), so i've had to do some backflips to get things to look clean in the UI might post something in the discourse, super cool what you can do with blocks + blocks api for not much code
a

Anna Geller

09/07/2022, 8:26 AM
Wow, this sounds really cool! It would be great if you could share it in any way, Discourse or even a GitHub Gist
j

Jeff Hale

10/31/2022, 6:59 PM
Hey @Alix Cook! Not sure exactly how you have things set up, but wanted to let you know that you could use
is_anonymous=True
when you create a DateTime Block so that you don’t need to create a named block. 🙂
a

Alix Cook

10/31/2022, 7:10 PM
we rely on the named blocks to be able to reference them in other flows. am i misunderstanding how anonymous blocks work?
j

Jeff Hale

10/31/2022, 7:35 PM
Gotcha. Makes sense.