Hello team! Trying to run a workflow with local dask executor, it works fine locally with prefect core and distributing over multiple threads or processes in my local machine over mapped tasks, but when I want to run the same workflow over prefect server, the prefect server runs those mapped tasks in sequential manner. Is this is an expected behavior? Or am I doing something wrong?
k
Kevin Kho
10/20/2021, 6:20 PM
Hey @Payam Vaezi , maybe it will help is you explicitly specify the number of threads for the local dask executor?
I tried explicitly defining those kwargs, but it seems prefect job disregards those values for some reason, and mapped tasks runs sequentially. Any idea what else may be wrong?
k
Kevin Kho
10/20/2021, 7:35 PM
Could you show me your flow code if it’s simple enough to share? I do have an idea
p
Payam Vaezi
10/21/2021, 8:09 PM
Thanks for follow up, just realized it was a user error on my side, once found that executor attribute added to correct flow object it was working fine. Thanks again for timely support.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.