Is there a way to dynamically change the properties of the flow executor?
I have a specific situation where I run say 30 tasks in parallel (using mapping) to retrieve data, but the data provider can reply that the API is overused in which case I need to reduce the amount of workers in the dask executor to 1, otherwise the provider will ban the IP.
I could handle that somehow internally to the download task with threading.Lock, but I guess that would only work for the threading local dask executor.
Can it be handled at the prefect flow or task level?
Aurélien Vallée
04/18/2021, 6:06 AM
Basically either I handle it with a lock of some sort inside the task, but then I need a lock mechanism that would work with whatever executor, which does not seem possible easily considering some workers are distributed. Or either I handle it by dynamically changing the executor workers from the flow.
nice, but it's prefect cloud only, and it's not dynamic from my understanding?
k
Kevin Kho
04/19/2021, 5:12 AM
Yes unfortunately this is only for Prefect cloud. I think your thought is right that there is not a good way to achieve this dynamic adjustment, especially with a parallel executor.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.