https://prefect.io logo
Title
j

Joshua Grant

11/22/2022, 6:47 PM
What is the Prefect 2 equivalent of
apply_map
? Details in 🧵
1
Previously flow looked like this:
from prefect import apply_map, Flow

from mycode import task1, task2, task3, get_org

def extract_from_file(file: dict, organization: str):
    task1_results = task1(file['key'])
    task2_results = task2(file['another_key'])
    task3_results = task3(file['yet_another_key'], organization)
    

with Flow('this-flow', ) as this_flow:
    files = Parameter('files')
    organization = get_org()
    flow_file = apply_map(
        extract_from_file,
        file=files,
        organization=unmapped(organization),
    )
z

Zanie

11/22/2022, 7:08 PM
Like this?
for file in files:
    task1.submit(file["key"])
    task2.submit(file["another_key"])
    etc...
🌮 1
I don’t think there’s a direct equivalent yet, since it most cases it’ll be more readable to just loop
j

Joshua Grant

11/22/2022, 7:09 PM
That'll work
Will this be run in parallel?
z

Zanie

11/22/2022, 7:19 PM
Yep!
🙌 2
(Concurrently in threads, technically)
If you want parallelism with multiprocessing you need to use a Dask task runner.