<@ULVA73B9P> hey in prefect cloud job run v2, can ...
# ask-marvin
a
@Marvin hey in prefect cloud job run v2, can we preserve the job run if it crashes
m
thought for 373.8 seconds
Short answer: if you’re using Prefect Cloud’s Job Run v2 on Prefect Serverless, no — the compute is ephemeral and we don’t expose a toggle to keep the container/pod around after a crash. Logs and run metadata are preserved in Prefect Cloud for debugging. If you need to preserve the underlying job for inspection, run on your own worker instead of Serverless and adjust the infrastructure settings: - Kubernetes worker: edit your work pool’s base job template to prevent automatic cleanup. - Omit
spec.ttlSecondsAfterFinished
(or set a large value) so Jobs aren’t deleted immediately - Optionally set
spec.backoffLimit: 0
to avoid retries and keep the first failed pod intact - Example base job template snippet:
Copy code
apiVersion: batch/v1
    kind: Job
    spec:
      backoffLimit: 0
      # omit ttlSecondsAfterFinished to avoid auto-cleanup, or set a large value
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: prefect-job
              # image/command/env are injected by the worker
- Configure this in Cloud: Work Pools -> your Kubernetes pool -> Base job template. - Docs: Kubernetes worker guide and Managing work pools - Docker worker: you can configure the base container settings to avoid auto-removal so failed containers stick around for inspection. If you’re on Docker, I can share an exact template. Regardless of compute, you can pull full logs for a failed run:
Copy code
prefect flow-run logs <flow-run-id>
CLI ref: prefect flow-run logs Can you confirm which compute you’re using for Job Run v2 — Prefect Serverless, a Kubernetes worker, or Docker? I’ll tailor the exact settings/template for your setup.
j
hey, that is not totally correct. There is a keep job toggle that the work pool exposes. However this is not controllable per state (like only Crashes). You can only set it on the work pool, deployment or flow run level as a job_variable
a
@Jake Kaplan hi, yes i was aware of this, but wanted to preserve the job, logs and metrics if the flow was not successful (my usecase being gcp sends a SIGKILL signal and i need to trace the logs/metrics for the root cause), or at-least control it inside the flow. Is there a workaround for this or any active issues on the same? or could you point me to the part of the codebase where i could contribute for this feature. thanks
👍 1
j
One workaround is you can keep every job and write an automation that will kick off a deployment to manually clean up successful jobs? I know that is obviously not ideal. But I'd definitely encourage you to open an issue for this here in the prefect repo! If you're looking to contribute the feature it would be in the cloud_run_v2 worker to add some extra configuration around when to clean up jobs or not outside of just yes/no.
🙌 1
a
thanks
👍 1