Thread
#prefect-community
    m

    Martim Lobao

    1 year ago
    hey all, I’ve also got a couple questions related to some issues we’ve encountered during some recent flow runs, not sure if this is the right place to ask for help:1. we have a flow which at one point has 3 tasks that are triggered in parallel when their dependent task finishes. for whatever reason, two of those tasks got triggered automatically as soon as the parent task finished, but the other one was stuck in a pending state. it eventually started, but only after the other 2 tasks finished. is there any reason why the 3 tasks did not run in parallel? 2. it can sometimes happen that we attempt to restart a flow but we have to click the restart button a couple of times before the actual flow restarts. is there any reason why this happens or is it just a prefect bug? this has happened on a couple of occasions now (here i clicked the restart button, waited 3 minutes and then clicked again, and didn’t change anything between attempts)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Martim Lobao, what executor are you using for number 1? Is it a mapped task run? Will ask the UI team for number 2
    m

    Martim Lobao

    1 year ago
    thanks @Kevin Kho, we’re using the
    LocalDaskExecutor
    for this particular flow
    Kevin Kho

    Kevin Kho

    1 year ago
    You can try explicitly stating:
    flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
    Are you specifying the
    num_workers
    ? Are you running on local or ECS? I have seen cases where local uses the CPU count (4) and then ECS just uses 2 by default.
    m

    Martim Lobao

    1 year ago
    hey, thanks Kevin, that was it 🙂
    ECS was using 2
    couple of questions: • is there any downside to increasing the number of workers to some absurd amount? even if there are resource limitations, i would expect that running tasks in series will never be faster than running them in parallel, given the same compute resources. • is there any reason to explicitly set
    scheduler="threads"
    ? this is the default value for the scheduler, right?
    Kevin Kho

    Kevin Kho

    1 year ago
    No reason on the threads. I literally copied that line from docs, which is why it’s there. I think there is a downside but I’m not sure. The flow might error by trying to simultaneously process too much
    m

    Martim Lobao

    1 year ago
    nice, thanks for the reply @Kevin Kho 🙂 lmk if you get any update regarding 2
    Kevin Kho

    Kevin Kho

    1 year ago
    Will follow up on that today