https://prefect.io logo
m

Muddassir Shaikh

01/19/2022, 5:35 PM
Hi here the tasks on Left branch are running fine but the tasks on the Right Branch are stuck in Mapped state. Both have the same functions called and mapped to run a task.
k

Kevin Kho

01/19/2022, 6:23 PM
Are you using LocalDaskExecutor?
m

Muddassir Shaikh

01/19/2022, 6:50 PM
Yes LocalDaskExecutor
k

Kevin Kho

01/19/2022, 6:51 PM
Are you using threads or processes? Processes might help it become more stable
m

Muddassir Shaikh

01/19/2022, 6:55 PM
Copy code
left_branch = processing.map(processing_channels([('some_var1', date)]))
right_branch =  processing.map(processing_channels([('some_var2', date)]))
k

Kevin Kho

01/19/2022, 6:56 PM
No no I mean
LocalDaskExecutor(scheduler="processes")
m

Muddassir Shaikh

01/19/2022, 6:57 PM
flow.executor = LocalDaskExecutor()
k

Kevin Kho

01/19/2022, 6:57 PM
Yes the default is
threads
.
processes
is more stable. You can see the syntax here
m

Muddassir Shaikh

01/19/2022, 6:58 PM
Thanks!
Also is there any maximum limit for parallel processing?
k

Kevin Kho

01/19/2022, 7:01 PM
Not on the Prefect side. It would be your hardware limits
m

Muddassir Shaikh

01/19/2022, 7:02 PM
Cool.
Hi @Kevin Kho i am still facing the Mapped state issue even after adding processes in LocalDaskExecutor
k

Kevin Kho

01/20/2022, 4:23 PM
Does it not run after the left side finishes running? Prefect currently only runs one mapped task at a time
m

Muddassir Shaikh

01/20/2022, 4:41 PM
Yes but the left side takes several hours to finish. I want to run both mapped task to run in parallel as they are not interdependent.
k

Kevin Kho

01/20/2022, 4:42 PM
Ah I see now. The FlowRunner currently only runs one mapped task at a time.
m

Muddassir Shaikh

01/20/2022, 4:46 PM
Ohh okay, let me know if you find any fix. Thanks!!