Newskooler

    Newskooler

    1 year ago
    Hi 👋 I am struggling to make my prefect depth-first and not breadth-first. My flow is set up like so:
    from prefect.executors import LocalDaskExecutor
    from prefect import Flow
    
    with Flow(name=name, executor=LocalDaskExecutor(scheduler="threads", num_workers=1)) as flow:
        ...
    Do I need to use another executer or where can I learn more about this?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Newskooler, this has to do with Dask preferring depth first execution, but it can’t be forced also and the scheduler will decide to do things in a breadth first way.
    Newskooler

    Newskooler

    1 year ago
    I remember before it used to always do depth first - can I make Prefect run in depth first in some way?
    Kevin Kho

    Kevin Kho

    1 year ago
    The only idea I have is try using
    DaskExecutor()
    instead of
    LocalDaskExecutor()
    , even if this is on Local. I thinkt he DaskExecutor tends more to be depth first from experience. You might get into issues with warnings about
    freeze_support()
    so you may have to do:
    if __name__ == "__main__":
        flow.run()
    If doing a local run.
    Newskooler

    Newskooler

    1 year ago
    So I don’t understand on what basis does it make a choice and why is that not a user choice?
    Kevin Kho

    Kevin Kho

    1 year ago
    So this is mainly a Dask question and how they choose which task to go next. It’s not something that’s in their docs, but it’s complex and based on heuristics for task size etc. It’s also not something you can predict a priori because it also matters. I know this answer may be unsatisfying to you, so I’ll go hunt down a better answer from people who know more and get back to you.
    Newskooler

    Newskooler

    1 year ago
    Okay - thanks a lot. I remember a few version back my code worked depth-first always and then smth changed.
    Kevin Kho

    Kevin Kho

    1 year ago
    That may be the Dask version and changes to their scheduler.
    Newskooler

    Newskooler

    1 year ago
    Cool thanks. Let me know if you find a way to do this 😃