Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey, Is there any documentation on how prefect and Kubernetes work together? It is a little unclear if all flows run get submitted to Kubernetes even if there is no capacity or if prefect queues these and submits the job when capacity exists,  If it is Perfect that manages this is it Kubernetes agent that deals with this? TIA

Hey <@U01720VJNSW>, Prefect starts the flow as Kubernetes jobs. If it can’t get the hardware needed, the flow will likely not start. You will see Lazarus resubmit the flow for execution. This commonly happens to users using Spot Instances for their Kubernetes cluster when they aren’t able to get those instances. You can read more about Lazarus <https://docs.prefect.io/orchestration/concepts/services.html#lazarus|here>

<@ULVA73B9P> open “Lazarus Docs say there are 10 retries but this should be 3”

<@U01QEJ9PP53> Are the number of retries configurable? and also is the time between retries configurable?

You can change <https://github.com/PrefectHQ/server/blob/master/src/prefect_server/config.toml#L78-L79|this> on server. Time between doesnt seem exposed

I think it’s <https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/lazarus.py#L25|here>

Also might be tied to the heartbeat cutoff  <https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/lazarus.py#L80|here>