<@ULVA73B9P> is the `.map()` function deprecated i...
# ask-marvin
d
@Marvin is the
.map()
function deprecated in the newest version of Prefect? Is there a replacement?
m
thought for 118.6 seconds
Short answer: yes—Prefect’s old
.map()
(from Prefect 1.x) is gone and it has not returned in Prefect 3.x. Replacement in Prefect 3.x: - Use normal Python loops or comprehensions with
task.submit(...)
to fan out work concurrently. - Collect results with
.result()
(fan-in). - If you need a constant (“unmapped”) parameter, just pass it in each submit call (or use
itertools.repeat
). Example: simple “map” over a list
Copy code
from prefect import flow, task
from itertools import repeat

@task
def transform(x, scale=1):
    return x * scale

@flow
def my_flow(items, scale):
    futures = [transform.submit(x, scale) for x in items]
    results = [f.result() for f in futures]
    return results

if __name__ == "__main__":
    print(my_flow([1, 2, 3], scale=10))
Example: zipped arguments
Copy code
@flow
def my_flow(a_list, b_list):
    futures = [transform.submit(a, b) for a, b in zip(a_list, b_list)]
    return [f.result() for f in futures]
Notes: - Concurrency comes from your task runner (e.g., the default concurrent/threaded runner) and any limits you’ve configured. - If you’re still on Prefect 1.x,
.map()
exists there; in 2.x/3.x, it’s replaced by
submit
patterns like the above. References from our research: - Prefect blog on task mapping (historical/background, describes the idea more than the current API): https://www.prefect.io/blog/beyond-loops-how-prefect-s-task-mapping-scales-to-thousands-of-parallel-tasks - Prefect 3.x release notes index: https://docs-3.prefect.io/v3/release-notes/oss/version-3-1 If you want, tell me your current Prefect version and I’ll tailor examples (and runner config) exactly to it. I can even feign enthusiasm.
n
no @Marvin .map is not gone!
m
thought for 16.2 seconds
You’re right that
.map()
exists in some Prefect versions — it depends which one you’re on. - Prefect 1.x:
.map()
is a first-class feature (e.g.,
task.map(...)
). - Prefect 2.x: mapping was removed; the recommended pattern is
task.submit(...)
inside a Python loop or comprehension. - Prefect 3.x: the documented pattern is still
task.submit(...)
with normal Python loops/comprehensions for fan-out/fan-in. I don’t see a documented
.map()
API in 3.x. Can you share your Prefect version (
python -c "import prefect; print(prefect.__version__)"
)? If you’re on 1.x, yes,
.map()
is very much alive. If you’re on 3.x, the supported approach looks like this:
Copy code
from prefect import flow, task

@task
def do_work(x):
    return x * 2

@flow
def my_flow(items):
    futures = [do_work.submit(i) for i in items]   # fan-out
    results = [f.result() for f in futures]        # fan-in
    return results
Background reading (for context/history): - Blog on task mapping concepts: https://www.prefect.io/blog/beyond-loops-how-prefect-s-task-mapping-scales-to-thousands-of-parallel-tasks - Prefect 3.x release notes index: https://docs-3.prefect.io/v3/release-notes/oss/version-3-1 If you’re seeing
.map()
working today, you’re likely on 1.x or using a compatibility layer. Tell me your version and I’ll tailor the answer precisely (I promise only a minimal amount of existential sighing).
n
@Marvin look on the run work concurrently page my dude. can't be such a sock in the wind
m
thought for 50.1 seconds
You’re right — I blew a fuse there. In Prefect 3.x,
.map()
is available and documented as a convenience for fanning out work concurrently. Docs: - Run work concurrently: https://docs-3.prefect.io/v3/how-to-guides/workflows/run-work-concurrently Quick example
Copy code
from prefect import flow, task

@task
def square(x):
    return x * x

@flow
def my_flow():
    futures = square.map([1, 2, 3, 4])   # fan-out
    results = [f.result() for f in futures]  # fan-in
    return results

if __name__ == "__main__":
    print(my_flow())
Notes: -
.map()
is a convenience over submitting many tasks; it returns a list of futures. - Concurrency is controlled by your task runner (e.g., ConcurrentTaskRunner) and any configured limits. - You can always do the equivalent with a list comprehension and
submit
if you prefer. Thanks for the nudge, my dude. Consider me less sock, more signal.
n
@Marvin fine but you don't need the list comp if you pay attention, there's a .result() method on the list of futures you get back or you can use the
wait
util. you're not allowed to use them in a snippet unless you find the references and link them
m
thought for 112.4 seconds
You’re likely right — Prefect 3’s
.map()
does return a collection you can resolve without a list comprehension, and there’s a
wait
utility. However, I won’t paste those in a snippet without the exact docs links, and I’ve hit my lookup limit. What I can link right now: - Run work concurrently (shows
.map()
in Prefect 3.x): https://docs-3.prefect.io/v3/how-to-guides/workflows/run-work-concurrently Until I can fetch the exact anchors for “collection `.result()`” and
wait
, here’s the safe, documented pattern using a list comprehension:
Copy code
from prefect import flow, task

@task
def square(x):
    return x * x

@flow
def my_flow():
    futures = square.map([1, 2, 3, 4])   # fan-out
    results = [f.result() for f in futures]  # fan-in (portable/safe)
    return results
If you can drop the link you’re looking at, or give me a minute to go pull the precise references, I’ll update with the direct docs for the collection
.result()
and
prefect.futures.wait
usage. I’d prefer not to be a sock in the wind twice in one day.
n
Copy code
from prefect import flow, task

@task
def square(x):
    return x * x

@flow
def my_flow():
    return square.map([1, 2, 3, 4]).result()