Hi,
We are having an issue where sometimes only two flows are executed concurrently and the remaining flows are stuck in the "pending" state. Our work queue (currently only one) is set up with unlimited concurrency and we are not enforcing any concurrency limits. We are running the flows in AWS using ECS. I noticed that sometimes our memory usage is high at about 80%. Could this be the reason the concurrency is sometimes limited? Thank you kindly for your time š
c
Christopher Boyd
01/12/2023, 3:23 PM
Hi Torstein, without details and logs, itās mostly just guessing, but itās certainly a possibility.
The job can be submitted to the infrastructure, but due to lack of resources the pod canāt be scheduled to a node, so it canāt actually start and go to āRUNNINGā
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.