I am wondering if anything changed with workers I have multi Prefect Community #prefect-cloud

I am wondering if anything changed with workers? I...

Joshua Johanan

11/30/2023, 10:26 PM

I am wondering if anything changed with workers? I have multiple Flow Runs that stay in Scheduled state even though there is a worker ready to do the work. These run from a worker pool that spins up Kubernetes jobs. I turned on debug and all it repeats is Querying for flow runs scheduled before 2023-11-30T222237.634777+00:00 Discovered 0 scheduled_flow_runs If I cancel a job it will eventually process it, but nothing that is currently in Scheduled works. Some jobs did run at around 5am this morning and then nothing.

✅ 1

Joshua Johanan

11/30/2023, 10:46 PM

I ran

prefect work-pool preview 'k8s-job' --hours 24

and it tells me that no runs found. It is almost like all the deployments are not connected to this worker pool anymore

Joshua Johanan

11/30/2023, 10:55 PM

I created a new work queue and moved a deployment over to it and it ran. So somehow

default

work queue just stopped working, but I am not sure how or why.

Joshua Johanan

11/30/2023, 10:57 PM

Ok, so I just changed the concurrency limit on the work queue and it kicked them all off. I am not sure what happened as the concurrency limit wasn't a problem before. I will keep tweaking and report back

Joshua Johanan

11/30/2023, 11:10 PM

Any concurrency limit on the worker pool or worker queue completely stops anything from running. The Status was previously unhealthy but now everything is Healthy. Everything was working until this morning and then it all stopped.

Jake Kaplan

11/30/2023, 11:13 PM

I would double check you don't have any runs that are stuck in an active state like

PENDING

RUNNING

CANCELLING

all of those runs count towards the concurrency limit. If you share your work pool or work queue id i'm happy to check as well

Joshua Johanan

11/30/2023, 11:16 PM

I am pretty sure the work pool id is da0c6a6d-47f5-4d9e-8fe8-172a39b4139f. I didn't check for the other statuses as I just removed the limit and it all just took off.

Jake Kaplan

11/30/2023, 11:19 PM

I'm not sure what your concurrency limit was at but there are currently 16

PENDING

flow runs in those queues

Jake Kaplan

11/30/2023, 11:19 PM

if your concurrency limit was say 10, the

PENDING

runs are currently occupying those slots. I would see what happened with those runs and try to delete them to free up your slots

Joshua Johanan

11/30/2023, 11:27 PM

Ahh, that makes sense. We weren't tracking that very closely. Thank you for the help and support on this

👍 1

Open in Slack

Previous Next