Is there a way to dynamically change the propertie...
# ask-communitya
Aurélien Vallée
04/18/2021, 6:00 AMIs there a way to dynamically change the properties of the flow executor?
I have a specific situation where I run say 30 tasks in parallel (using mapping) to retrieve data, but the data provider can reply that the API is overused in which case I need to reduce the amount of workers in the dask executor to 1, otherwise the provider will ban the IP.
I could handle that somehow internally to the download task with threading.Lock, but I guess that would only work for the threading local dask executor.
Can it be handled at the prefect flow or task level?
Aurélien Vallée
04/18/2021, 6:06 AMBasically either I handle it with a lock of some sort inside the task, but then I need a lock mechanism that would work with whatever executor, which does not seem possible easily considering some workers are distributed. Or either I handle it by dynamically changing the executor workers from the flow.
k
Kevin Kho
04/18/2021, 2:59 PM
Hi @Aurélien Vallée! Task concurrency limiting is the best way to do this: https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html . Just set the limit below the API usage limit
a
Aurélien Vallée
04/19/2021, 4:00 AMnice, but it's prefect cloud only, and it's not dynamic from my understanding?
k
Kevin Kho
04/19/2021, 5:12 AM
Yes unfortunately this is only for Prefect cloud. I think your thought is right that there is not a good way to achieve this dynamic adjustment, especially with a parallel executor.
7 Views