Guy Propper

    Guy Propper

    11 months ago
    Hi, I have a question about running two mapped tasks one after the other. I have the following flow:
    file_paths_to_process_list = extract_files_to_process()
    mapped_parsed_output = transform.map(file_paths_to_process_list)
    load_output_to_db.map(mapped_parsed_output)
    I want map#2 (load_output_to_db) to depend on results from map#1 (transform), but not to wait for all results from map#1. As soon as there is one result from map#1, it should be processed in map#2. What is the correct way to do this? Thanks!
    Henning Holgersen

    Henning Holgersen

    11 months ago
    Commenting mostly to follow the thread and see possible solutions, but my intuitive thought is to call
    transform
    from within the
    extract_files_to_process
    and pass the return value on.
    Kevin Kho

    Kevin Kho

    11 months ago
    What executor are you using?
    Guy Propper

    Guy Propper

    11 months ago
    @Kevin Kho what do you mean? I am new to Prefect so I might not be using it the right way
    Kevin Kho

    Kevin Kho

    11 months ago
    You can specify a LocalDaskExecutor or DaskExecutor. The default is a sequential LocalExecutor. Docs for this can be found here. I ask because the DaskExecutor does depth-first execution. LocalExecutor isn’t really configurable. Maybe you can try changing to LocalDaskExecutor and see if it helps?
    flow.executor = LocalDaskExecutor()