https://prefect.io
Join Slack
Hi here the tasks on Left branch are running fine but the tasks on the Right Branch are stuck in Map...
# ask-community
m
Hi here the tasks on Left branch are running fine but the tasks on the Right Branch are stuck in Mapped state. Both have the same functions called and mapped to run a task.
k
Are you using LocalDaskExecutor?
m
Yes LocalDaskExecutor
k
Are you using threads or processes? Processes might help it become more stable
m
Copy code
left_branch = processing.map(processing_channels([('some_var1', date)]))
right_branch =  processing.map(processing_channels([('some_var2', date)]))
k
No no I mean 
LocalDaskExecutor(scheduler="processes")
m
flow.executor = LocalDaskExecutor()
k
Yes the default is 
threads
. 
processes
is more stable. You can see the syntax here
m
Thanks!
Also is there any maximum limit for parallel processing?
k
Not on the Prefect side. It would be your hardware limits
m
Cool.
Hi @Kevin Kho i am still facing the Mapped state issue even after adding processes in LocalDaskExecutor
k
Does it not run after the left side finishes running? Prefect currently only runs one mapped task at a time
m
Yes but the left side takes several hours to finish. I want to run both mapped task to run in parallel as they are not interdependent.
k
Ah I see now. The FlowRunner currently only runs one mapped task at a time.
m
Ohh okay, let me know if you find any fix. Thanks!!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by