hey all, I’ve also got a couple questions related ...
# ask-community
m
hey all, I’ve also got a couple questions related to some issues we’ve encountered during some recent flow runs, not sure if this is the right place to ask for help: 1. we have a flow which at one point has 3 tasks that are triggered in parallel when their dependent task finishes. for whatever reason, two of those tasks got triggered automatically as soon as the parent task finished, but the other one was stuck in a pending state. it eventually started, but only after the other 2 tasks finished. is there any reason why the 3 tasks did not run in parallel? 2. it can sometimes happen that we attempt to restart a flow but we have to click the restart button a couple of times before the actual flow restarts. is there any reason why this happens or is it just a prefect bug? this has happened on a couple of occasions now (here i clicked the restart button, waited 3 minutes and then clicked again, and didn’t change anything between attempts)
k
Hi @Martim Lobao, what executor are you using for number 1? Is it a mapped task run? Will ask the UI team for number 2
m
thanks @Kevin Kho, we’re using the
LocalDaskExecutor
for this particular flow
k
You can try explicitly stating:
Copy code
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
Are you specifying the
num_workers
? Are you running on local or ECS? I have seen cases where local uses the CPU count (4) and then ECS just uses 2 by default.
m
hey, thanks Kevin, that was it 🙂
ECS was using 2
couple of questions: • is there any downside to increasing the number of workers to some absurd amount? even if there are resource limitations, i would expect that running tasks in series will never be faster than running them in parallel, given the same compute resources. • is there any reason to explicitly set
scheduler="threads"
? this is the default value for the scheduler, right?
k
No reason on the threads. I literally copied that line from docs, which is why it’s there. I think there is a downside but I’m not sure. The flow might error by trying to simultaneously process too much
m
nice, thanks for the reply @Kevin Kho 🙂 lmk if you get any update regarding 2
k
Will follow up on that today
🙌 1