Raphaël Riel
10/14/2020, 12:44 PMthe_task.map(list_of_ints)
. The Task will have to Map >10 items.
When A) running the flow directly from within the .py file using flow.run()
AND B) setting executor=LocalDaskExecutor()
I’m able to have it use more than 1 thread.
But as soon as I try to run this Flow in an Agent OR if I remove the Dask Executor (While executing the .py file directly), I can’t make it run in parallel!
Recap:
1. Execute the flow from .py file WITH executor=LocalDaskExecutor()
= Works
2. Execute the flow from .py file with “default” Executor = Nope
3. Any combinaison of executor running in an agent = Nope
Any suggestion will be welcome! Thanks.josh
10/14/2020, 12:57 PMRaphaël Riel
10/14/2020, 1:05 PMjosh
10/14/2020, 2:31 PM@task
def vals():
return [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
@task
def printv(v):
print(v)
with Flow("local-dask", environment=LocalEnvironment(executor=LocalDaskExecutor())) as f:
v = vals()
printv.map(v)
I see the tasks being executed in non-sequential order. I wonder if it has something to do with Dask thinking you only have one thread available. What happens when you set your executor to something like LocalDaskExecutor(nthreads=4)
?Raphaël Riel
10/14/2020, 2:37 PMjosh
10/14/2020, 2:39 PMRaphaël Riel
10/14/2020, 2:45 PMjosh
10/14/2020, 2:47 PMRaphaël Riel
10/14/2020, 2:50 PM