Hi all wonder if anyone has any experience running prefect f

Question

Hi all wonder if anyone has any experience running prefect flows on GKE spot instances Is there any way of dealing gracefully with possible SIGTERM signals

Anna Geller · Answer

Do you mean GKE or really I think Autopilot should work fine without you having to worry about the SIGTERM signals

Kevin Kho · Answer

There is an open issue for the SIGTERM <https github com PrefectHQ prefect issues 5198|here> I think that is your use case right

Joshua Greenhalgh · Answer

So I mean spot on autopilot <https cloud google com kubernetes engine docs how to autopilot spot pods>

Joshua Greenhalgh · Answer

< Kevin Kho> that is exactly what I am after my understanding is that what will happen is that cloud will just think tasks are running but never terminate if the spot pod gets killed

Joshua Greenhalgh · Answer

I accept that if my jobs are pretty short its very unlikely to happen but still may do

Kevin Kho · Answer

It won t terminate because there is no signal telling Prefect Cloud to end it but on the Cloud side if there is no heartbeat obtained from the Flow it will eventually be marked as failed

Anna Geller · Answer

Joshua I m curious if you estimated thought about whether it s worth the costs and when it s worth it It seems the only difference between Autopilot and Autopilot spot is the price And given that it s already so affordable I wonder whether your time is worth more smile I hope it doesn t sound negative in any way but I m just curious about the number of flows this cost benefit of Spot Autopilot becomes worth the extra management overhead

Anna Geller · Answer

btw so timely my blog post got released just now on smile

Joshua Greenhalgh · Answer

~Its 50 less 50 decrease in costs is 50 decrease in costs ~

Joshua Greenhalgh · Answer

As an example for my very minimal needs Compute 2hr per day ~8 util avg 24 vCPU avg 32GB mem the diff in cost is $27 per month vs $92 yeah not huge but proportionally a lot

Joshua Greenhalgh · Answer

And having a nice why to deal with the possible termination is worth it for me

Joshua Greenhalgh · Answer

If we were talking significantly more scale than me it becomes even more worth it no

Joshua Greenhalgh · Answer

Like its almost 3x less

Joshua Greenhalgh · Answer

Nice article by the way I think for most prefect like usercases ETL stuff cold starts is really no an issue so not even something I have worried about if I want my process to start at 6am and they start at 6 01 thats fine by me

Anna Geller · Answer

It s likely that your hourly rate is higher than those monthly cost savings smile I think those cost savings are so minimal do you think they are worth the management overhead Comparing in ratio makes little sense with such small numbers but you should decide for yourself and I can definitely understand if you have some restrictions imposed upon you from above Glad you liked the article LMK if there are any questions left I can help with

Joshua Greenhalgh · Answer

You are right from a money perspective for my usecase of course but there are constraints from above as always the ability to know that prefect will just do the right thing on possible termination is still worth it though I think for prefect if we were talking much much more spend then being able to reduce that by 3x just cos there is SIGTERM handler would I think be of huge benefit

Anna Geller · Answer

Gotcha I only think that looking at it in ratio cost benefit is not helpful when looking at such small amounts of cost savings Compute is cheap these days engineering time is expensive Regarding the SIGTERM handling I don t think we will tackle this in Prefect 1 0 but in Prefect 2 0 we do have a `Crashed` state already so this may be handled more gracefully in the future