Roger Webb

04/11/2022, 1:43 PM
Using the LocalDaskExecutor... is there a way to limit the number of tasks to a specific number? With the schematic that a flow shows based on dependancies, We need to limit the number of executions in the entire flow to only be running 20 tasks at a time. Is there a configuration that Im not seeing?

Kevin Kho

04/11/2022, 1:47 PM
Hi @Roger Webb, 20 of the same task or 20 tasks across all the tasks? For the same task, there is task run concurrency limits and for the global, you can just limit it on the Executor level:
flow.executor = LocalDaskExecutor(num_workers=20)
as seen here