Hello <@ULVA73B9P> Is there a native way to kill/s...
# ask-marvin
s
Hello @Marvin Is there a native way to kill/stop Prefect Job (kubernetes pod) from execution after 8 hours of running?
m
I found some references in the Prefect documentation that might be relevant to controlling the runtime of tasks. While I could not find a direct native method to kill a Kubernetes pod that belongs to a Prefect job after exactly 8 hours, there's information on an improved cancellation setting that you can experiment with: 1. Enhanced cancellation: Prefect is working on improving cases where cancellation can fail. There's an experimental feature that can be enabled by setting
PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION=True
on your worker or agents. This may help in ensuring that cancellation requests are more reliably executed, but it doesn't enforce a strict timeout. You can find more information here (Search for "Enhanced cancellation"). Unfortunately, the above references do not provide a direct solution. In general, task timeout management could be handled at the Kubernetes level using a
CronJob
to execute a script that checks the runtime of pods and kills those exceeding the specified limit, or by writing custom logic in your Python code to enforce a timeout. If you'd like to implement a custom solution in Python, you could use a Prefect
task
decorator with a
timeout
parameter, but this would apply on the task level, rather than the whole job. Adjusting this to fit a Kubernetes deployment might require extra configuration or custom code. Additionally, it might be beneficial to look into any updates in Prefect's release notes or discuss this requirement with the Prefect community on their Slack channel or GitHub repository. If you need help with that or have more questions, feel free to ask.