Is there a way to have a Prefect GPU agent/worker on (temporary) sleep when it’s not used in Kubernetes?Motivation: let’s assume that the cost of a heavy GPU machine is $2k per day.
Let’s also assume that our Prefect deployment runs in AKS (managed Azure Kubernetes) and we have separate pods for
prefect-server
and a
prefect-agent
.
What if 90% of the day the
prefect-agent
(which is running on a GPU node on the cluster) is idle?
This means it’s underutilized and we waste money for no good reason.
Reference: Airflow provides Kubernetes Executor - on-demand/ad-hoc worker pods.
Since Prefect thought of everything - I’m sure there is either a built-in capability for that or a design pattern for achieving that.
Thanks!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.