Aniruddha Sengupta

    Aniruddha Sengupta

    6 months ago
    Hello Hope that you are well. I have a parent task I have created and I am able to create a create a series of task runs based on the inputs according to this documentation: https://docs.prefect.io/core/idioms/task-run-names.html. How can I ensure that the children tasks that are created from this are able to run in parallel instead of sequentially? Do I need to set the flow's executor to be the LocalDaskExecutor? Thanks for your help,
    Kevin Kho

    Kevin Kho

    6 months ago
    Hi @Aniruddha Sengupta, yes exactly you need to use the LocalDaskExecutor for parallelization
    Aniruddha Sengupta

    Aniruddha Sengupta

    6 months ago
    Hi Kevin Thanks for your reply. Just to clarify, I have a task called
    url_extraction
    which takes a
    lang
    input. When the Flow is then created, it is mapped to a list of languages which creates a series of children tasks based on the language. When I set the Flow's executor like so:
    flow.executor = LocalDaskExecutor()
    , this will ensure that the children tasks will be executed in parallel? Thanks, Aniruddha
    Kevin Kho

    Kevin Kho

    6 months ago
    If they are independent, yes
    Aniruddha Sengupta

    Aniruddha Sengupta

    6 months ago
    Ok, thanks so much for your help Kevin.
    Kevin Kho

    Kevin Kho

    6 months ago
    Of course!