A number of flows in prefect cloud are stuck in submitted and zombie killer is not putting them to bed, any suggestions on getting them processed?

Hi @George Coyne - that doesn't sound out of the ordinary, zombie killer is only responsible for dealing with tasks that are running that haven't issued a heartbeat in the last 2 minutes. For flows stuck in a submitted state, the lazarus process is responsible for re-scheduling those every ~6 minutes or so, to a maximum of 10 reschedules

Ah true

Hey @George Coyne what’s your new agent version?

0.12.6

We’re checking one thing on our end, just a moment

Flow run logs are good, running on k8s, it just looks like some jobs aren’t actually queueing or something? I’m digging into it a little deeper but wanted to bring it to you guys sooner rather than later

Hey @George Coyne just to double check, you’re saying that the first time the run is submitted the agent doesn’t create a k8s job but the second time the run goes into a submitted state the agent properly creates a job?

Inconsistently but yes, some flows get pulled perfectly, some just don’t. I can’t determine any consistency to the behavior.

In the logs for the flow run, there should be a job ID for the created job. Take a look at the jobs created with those IDs. What state are the jobs in? Are they “Unschedulable”?

That’s the thing, the jobs just don’t appear

🧐

prefect-job-8ffbfe27

for instance, stuck in submitted since 11:00 cdt

Would you shoot me that flow run ID?

Is that flow run present in your agent logs?

Also one thing to confirm when checking for the job:

kubectl get jobs --all-namespaces

The fact that it’s stuck in submitted with a job ID attached means the agent is submitting it to k8s 🤔

I do have a second agent running

ah, so it’s not picking up runs accidentally?

No jobs of in either cluster as of right now

Both agents will pick up flows unless you add labels to both agents and flows

Yep I love that too, but in this situation both are running without a label

Gotcha, so you’re just trying to get an agent to submit a flow run?

Agents are in seperate clusters

Is the new agent performing better?

I am trying to understand, and stop, flows from getting submitted without running

Understood

Nope