Hey there! I have a really weird issue with Kubernetes Jobs that I can’t wrap my head around: I try to start around 150 flows from on orchestrator flow. I do this because I need to parallelize across K8s pods instead of threads. This works, but only around 50 of my Flow-Runs actually transition into

Running

state and start a K8s job & pod, the other 100 Flow Runs just idle around in “Pending”. I have searched everywhere to find hints about why they won’t start. There is definitely enough K8s computing resources for them to be scheduled, there is no concurrency limits set via tags or on the work queue directly. The K8s work queue just shows them as “Pending” in the UI. Does anyone have any idea where else to look? Cheers 🙂

does kubectl describe pods *${POD_NAME}* reveal anything?

There is no pods (or jobs) on K8s yet, that’s the problem. Prefect doesn’t even try to start the jobs it seems

Ah sorry I thought you meant pending pod

You are completely right, I described that very badly. Fixed it now. Thanks a lot 🙂

🙂

Yes I do

Hmmm odd number - I have not tried to spin up as many flows at once as that. Just wondering if you need to shard it across agents, does the agent log report anything

unfortunately it logs everything that every single pod logs so it’s pretty hard to get any useful information from it 😕

I am not sure if two agents to one queue distribute the work semi evenly or does first one to poll grab 'all' the work - you may need to set the concurrency on the work-queue to stop this from happening and split the load across n agents

I have just a single agent instance running so this shouldn’t be a problem I think 🤔 I also tried to fix the work queue concurrency limit to 200 but it didn’t change anything. I have this fear that it’s somehow related to rate limiting on the free Prefect Cloud Tier but I have no idea how to verify this

Looking at the agent code it gets just 10 flows each time it polls so you could run another agent and see if you get over 52 concurrent - I have not seen any word about limits 🤷‍♂️

The problem is that then I’ll also need to change the DB because it’s currently using the SQLite one 😕

the agents are independent of the orion db backed by sqllite though?

Are you sure about that? 🤔 I just saw the following line in the agent manifest on discourse:

replicas: 1 _# We're using SQLite, so we should only run 1 pod_

how many max connections do you have on the database

@Christopher Boyd Thanks, that’s very helpful! Is there any way to validate that it’s definitely the SQLite limiting concurrency and no other factor? I only want to avoid adding new infrastructure to our stack only to end up with the same problem again :D

sorry, I didn’t realize you were using prefect cloud, they are separate

it’s very interesting, the log show that the Agent keeps trying to schedule the same flow run again and again:

13:37:34.302 | DEBUG   | prefect.agent - Checking for flow runs...
13:37:34.524 | INFO    | prefect.agent - Submitting flow run '<SAME_FLOW_RUN_ID>'

This keeps re-appearing 100 times

Do you have concurrency limits set at all? https://docs.prefect.io/concepts/tasks/#task-run-concurrency-limits

I do but they are all way lower (5, 10) so it couldn’t be them I think, since I see ~40-50 pods actually spawn up

at risk of the obvious, what happens if you set the limit to something like 100

Nothing unfortunately ;(