I have a strange situation with Prefect Cloud.
When I run a flow it is not able to start - remains in the late state.
I have a single work-pool with concurrency set to 1 and a default queue with unlimited concurrency. The worker is running on Kubernetes.
if I change the concurrency to 10 on the work pool it starts processing a single flow at a time. If I increase it even further to 1000 it can process a few more at a time.
I would expect that setting the concurrency level to N should allow for N concurrent flows to be executed at once.
When running on a local server, it behaves as expected.
f
Frits De Roos
05/06/2024, 3:28 PM
I had this same issue. Solution for me was to replace my (deprecated) agent-pool with a worker pool.
More info on deprecation here:
https://docs.prefect.io/latest/api-ref/prefect/agent/
Edit: my guess is that this weekend Prefect Cloud has had an update and that since Prefect Agents aren't working as before.
m
Maciej Kluczny
05/07/2024, 7:47 AM
Thanks for the reply, actually I use a worker pool right now with a worker running on Kubernetes.
👍 1
f
Frits De Roos
05/07/2024, 9:41 AM
Ahh different situation then. Hope you find a solution
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.