ENV

prefect==2.6.4 python==3.10 We have a

for loop

that uses the

prefect.deployments

python library to create deployment runs https://docs.prefect.io/api-ref/prefect/deployments/?h=run_deplo#prefect.deployments.run_deployment The deployment uses s3 storage and k8 job infrastructure. This flow run status error only seems to show up on some flow runs, while others work just fine. Our k8 cluster doesn't appear to be overloaded (it was at first, and we thought that was the issue, but we still get this error at normal load levels) We are exploring if maybe this is because flow runs in the for loop are using the same flow run names, and therefore trying to make duplicate jobs, and confusing the k8 agent.

I believe I have the same issue here https://github.com/PrefectHQ/prefect/issues/7116 It sounds like our k8 is rescheduling pods, and in doing so, the new pod fails to run because the flow run state in prefect is already running. this seems like a big issue, unless there is a better way to spec our job infrastructure. below should be the default spec generated by the CLI (minus the image secret)

{
  "kind": "Job",
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "env": [],
            "name": "prefect-job"
          }
        ],
        "completions": 1,
        "parallelism": 1,
        "restartPolicy": "Never",
        "imagePullSecrets": [
          {
            "name": "dockercloud-secret"
          }
        ]
      }
    }
  },
  "metadata": {
    "labels": {}
  },
  "apiVersion": "batch/v1"
}