It turns out that the flow was scheduled, submitted and run 2x by agent in the same flow run, causing this. Any reason why this happened?

Also in (kubernetes) agent logs I see that it’s experiencing read timeouts like this one quite often when trying to start flow run

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)

it’s not really visible, but in the logs, the same tasks was started 2x

with several seconds between starts

Hi @Kevin Kho 🙂 I just wanted to see if there are any updates on what might be the cause of this? Thanks!

or you meant the detailed view

is this a good practice? How many do you recommend?

We plan to run more of them which would spawn jobs in different namespaces for different teams with labels set accordingly. But for now we have just this one.

Ok let me try to get input from the team again. It is. We really just recommend one

Thanks 🙂

which would spawn jobs

because all our flows are k8s runs (

KubernetesRun

) (for

run_config

)

Prefect Cloud has a mechanism to stop these tasks from re-running which is called Version Locking but it is not available on server

Hello again. I checked this issue and the PR that was merged and I’m not sure this will fix the issue as it’s related to the running by duplicate agents (and the timespan of check is 30 seconds). The problem we have is related to Lazarus project which reschedules the flow and doesn’t kill the previous one. Example from today is below. GraphQL request:

query { flow_run(
  where: { 
    flow: { name: { _eq: "<flow-name>" } }
    scheduled_start_time: { _eq: "2021-09-28T05:10:00" }
  }) {
    flow {
      name
    }
    id
		states {
      state
      message
      created
      start_time
    }
  	state
    scheduled_start_time
		auto_scheduled
  }
}

Response:

{
  "data": {
    "flow_run": [
      {
        "flow": {
          "name": "<name>"
        },
        "id": "<id>",
        "states": [
          {
            "state": "Success",
            "message": "All reference tasks succeeded.",
            "created": "2021-09-28T05:36:09.345878+00:00",
            "start_time": null
          },
          {
            "state": "Failed",
            "message": "Kubernetes Error: pods ['prefect-job-<job-id>'] failed for this job",
            "created": "2021-09-28T05:35:56.861441+00:00",
            "start_time": null
          },
          {
            "state": "Running",
            "message": "Running flow.",
            "created": "2021-09-28T05:25:45.80988+00:00",
            "start_time": null
          },
          {
            "state": "Submitted",
            "message": "Submitted for execution",
            "created": "2021-09-28T05:25:42.091069+00:00",
            "start_time": null
          },
          {
            "state": "Scheduled",
            "message": "Rescheduled by a Lazarus process.",
            "created": "2021-09-28T05:20:08.821885+00:00",
            "start_time": "2021-09-28T05:20:08.815652+00:00"
          },
          {
            "state": "Submitted",
            "message": "Submitted for execution",
            "created": "2021-09-28T05:10:00.035923+00:00",
            "start_time": null
          },
          {
            "state": "Scheduled",
            "message": "Flow run scheduled.",
            "created": "2021-09-27T19:10:28.001225+00:00",
            "start_time": "2021-09-28T05:10:00+00:00"
          }
        ],
        "state": "Success",
        "scheduled_start_time": "2021-09-28T05:10:00+00:00",
        "auto_scheduled": true
      }
    ]
  }
}

Is there any suggestion how to resolve this? Thanks! 🙂

Please feel free to follow up faster if I don’t get back to in like 1 or 2 days. No need to wait a week