https://prefect.io
Join Slack
What is the Prefect 2 equivalent of `apply_map`? Details in :thread:
# ask-community
j
What is the Prefect 2 equivalent of 
apply_map
? Details in 🧵
✅ 1
Previously flow looked like this:
Copy code
from prefect import apply_map, Flow

from mycode import task1, task2, task3, get_org

def extract_from_file(file: dict, organization: str):
    task1_results = task1(file['key'])
    task2_results = task2(file['another_key'])
    task3_results = task3(file['yet_another_key'], organization)
    

with Flow('this-flow', ) as this_flow:
    files = Parameter('files')
    organization = get_org()
    flow_file = apply_map(
        extract_from_file,
        file=files,
        organization=unmapped(organization),
    )
z
Like this?
Copy code
for file in files:
    task1.submit(file["key"])
    task2.submit(file["another_key"])
    etc...
🌮 1
I don’t think there’s a direct equivalent yet, since it most cases it’ll be more readable to just loop
j
That'll work
Will this be run in parallel?
z
Yep!
🙌 2
(Concurrently in threads, technically)
If you want parallelism with multiprocessing you need to use a Dask task runner.
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by