Hi, I have a question about running two mapped tasks one after the other. I have the following flow:

file_paths_to_process_list = extract_files_to_process()
mapped_parsed_output = transform.map(file_paths_to_process_list)
load_output_to_db.map(mapped_parsed_output)

I want map#2 (load_output_to_db) to depend on results from map#1 (transform), but not to wait for all results from map#1. As soon as there is one result from map#1, it should be processed in map#2. What is the correct way to do this? Thanks!

Commenting mostly to follow the thread and see possible solutions, but my intuitive thought is to call

transform

from within the

extract_files_to_process

and pass the return value on.

What executor are you using?

@Kevin Kho what do you mean? I am new to Prefect so I might not be using it the right way

You can specify a LocalDaskExecutor or DaskExecutor. The default is a sequential LocalExecutor. Docs for this can be found here. I ask because the DaskExecutor does depth-first execution. LocalExecutor isn’t really configurable. Maybe you can try changing to LocalDaskExecutor and see if it helps?

flow.executor = LocalDaskExecutor()