In one of my flows, I'm using a concurrency limit of 1 on a particular resource.
This is working, but I've noticed that the relevant task is only being executed once every 30 seconds or so. The rest of the time my flow is just sitting idle.
If I inspect the concurrency-limit via the CLI while my flow is running, "Active Task Run IDs" is empty most of the time.
Here is a gist that demonstrates this. With a concurrency of N, you get batches of N tasks running with a 30 second delay between each batch. My case is just with N == 1.
Is there some kind of polling interval built into prefect, such that concurrency limits are only checked once every 30 seconds? If so, is there any way to reduce this delay? Is this behaviour documented anywhere?
j
Jeff Hale
03/30/2023, 1:16 PM
The default query interval should be 10 seconds, but you can change that via environment variable. See Settings here.
d
David Salgado
04/03/2023, 1:52 AM
Thanks for that. I'd recommend mentioning this in the documentation here
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.