Hello
Hope that you are well.
I have a parent task I have created and I am able to create a create a series of task runs based on the inputs according to this documentation: https://docs.prefect.io/core/idioms/task-run-names.html. How can I ensure that the children tasks that are created from this are able to run in parallel instead of sequentially? Do I need to set the flow's executor to be the LocalDaskExecutor?
Thanks for your help,
k
Kevin Kho
03/08/2022, 2:20 PM
Hi @Aniruddha Sengupta, yes exactly you need to use the LocalDaskExecutor for parallelization
a
Aniruddha Sengupta
03/08/2022, 2:43 PM
Hi Kevin
Thanks for your reply.
Just to clarify, I have a task called
url_extraction
which takes a
lang
input. When the Flow is then created, it is mapped to a list of languages which creates a series of children tasks based on the language.
When I set the Flow's executor like so:
flow.executor = LocalDaskExecutor()
, this will ensure that the children tasks will be executed in parallel?
Thanks,
Aniruddha
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.