Using the LocalDaskExecutor... is there a way to limit the number of tasks to a specific number? With the schematic that a flow shows based on dependancies, We need to limit the number of executions in the entire flow to only be running 20 tasks at a time. Is there a configuration that Im not seeing?
k
Kevin Kho
04/11/2022, 1:47 PM
Hi @Roger Webb, 20 of the same task or 20 tasks across all the tasks? For the same task, there is task run concurrency limits and for the global, you can just limit it on the Executor level:
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.