https://prefect.io
Join Slack
Hi all - wonder if anyone has any experience running prefect flows on GKE spot instances? Is there a...
# ask-community
j
Hi all - wonder if anyone has any experience running prefect flows on GKE spot instances? Is there any way of dealing gracefully with possible SIGTERM signals?
a
Do you mean GKE autopilot or really Spot VMs? I think Autopilot should work fine without you having to worry about the SIGTERM signals
k
There is an open issue for the SIGTERM here. I think that is your use case right?
j
@Kevin Kho that is exactly what I am after - my understanding is that what will happen is that cloud will just think tasks are running but never terminate if the spot pod gets killed?
I accept that if my jobs are pretty short its very unlikely to happen but still may do...
k
It won’t terminate because there is no signal telling Prefect Cloud to end it, but on the Cloud side, if there is no heartbeat obtained from the Flow, it will eventually be marked as failed.
👍 1
a
Joshua, I'm curious if you estimated/thought about whether it's worth the costs and when it's worth it? It seems the only difference between Autopilot and Autopilot spot is the price. And given that it's already so affordable I wonder whether your time is worth more 😄 I hope it doesn't sound negative in any way, but I'm just curious about the number of flows this cost-benefit of Spot Autopilot becomes worth the extra management overhead.
🎉 1
j
Its 50% less - 50% decrease in costs is 50% decrease in costs!
As an example for my very minimal needs (Compute - 2hr per day (~8% util) avg 24 vCPU, avg 32GB mem) - the diff in cost is $27 per month vs $92 - yeah not huge but proportionally a lot!
And having a nice why to deal with the possible termination is worth it for me
If we were talking significantly more scale than me it becomes even more worth it no?
Like its almost 3x less
Nice article by the way - I think for most prefect like usercases (ETL stuff) cold starts is really no an issue so not even something I have worried about - if I want my process to start at 6am and they start at 6:01 thats fine by me
👍 1
a
It's likely that your hourly rate is higher than those monthly cost savings 😄 I think those cost savings are so minimal - do you think they are worth the management overhead? Comparing in % ratio makes little sense with such small numbers, but you should decide for yourself and I can definitely understand if you have some restrictions imposed upon you from above. Glad you liked the article, LMK if there are any questions left I can help with
j
You are right from a money perspective (for my usecase) of course!! but there are constraints from above as always - the ability to know that prefect will just do the right thing on possible termination is still worth it though I think for prefect - if we were talking much much more spend then being able to reduce that by 3x just cos there is SIGTERM handler would I think be of huge benefit!
a
Gotcha. I only think that looking at it in % ratio cost-benefit is not helpful when looking at such small amounts of cost savings. Compute is cheap these days, engineering time is expensive. Regarding the SIGTERM handling, I don't think we will tackle this in Prefect 1.0, but in Prefect 2.0 we do have a 
Crashed
state already so this may be handled more gracefully in the future.
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by