hey all, I’ve also got a couple questions related to some issues we’ve encountered during some recent flow runs, not sure if this is the right place to ask for help:
1. we have a flow which at one point has 3 tasks that are triggered in parallel when their dependent task finishes. for whatever reason, two of those tasks got triggered automatically as soon as the parent task finished, but the other one was stuck in a pending state. it eventually started, but only after the other 2 tasks finished. is there any reason why the 3 tasks did not run in parallel?
2. it can sometimes happen that we attempt to restart a flow but we have to click the restart button a couple of times before the actual flow restarts. is there any reason why this happens or is it just a prefect bug? this has happened on a couple of occasions now (here i clicked the restart button, waited 3 minutes and then clicked again, and didn’t change anything between attempts)
k
Kevin Kho
09/08/2021, 5:34 PM
Hi @Martim Lobao, what executor are you using for number 1? Is it a mapped task run? Will ask the UI team for number 2
? Are you running on local or ECS? I have seen cases where local uses the CPU count (4) and then ECS just uses 2 by default.
m
Martim Lobao
09/09/2021, 11:43 AM
hey, thanks Kevin, that was it 🙂
Martim Lobao
09/09/2021, 11:43 AM
ECS was using 2
Martim Lobao
09/09/2021, 11:45 AM
couple of questions:
• is there any downside to increasing the number of workers to some absurd amount? even if there are resource limitations, i would expect that running tasks in series will never be faster than running them in parallel, given the same compute resources.
• is there any reason to explicitly set
scheduler="threads"
? this is the default value for the scheduler, right?
k
Kevin Kho
09/09/2021, 2:01 PM
No reason on the threads. I literally copied that line from docs, which is why it’s there. I think there is a downside but I’m not sure. The flow might error by trying to simultaneously process too much
m
Martim Lobao
09/09/2021, 2:17 PM
nice, thanks for the reply @Kevin Kho 🙂
lmk if you get any update regarding 2
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.