Is there an elegant way to control the level of concurrency using the
map()
functionality? My scenario is a list of URLs that should be used for making requests to but want to perform the requests in batches of 'n' rather than overwhelming a server with them all at once.
a
Anna Geller
01/13/2022, 11:44 AM
Yes, there is - you can leverage tags on your task decorator to set the tag, and then in the UI you can leverage task run concurrency to limit the number of concurrent task runs for that tag.
Copy code
from prefect import task
@task(tags=["yourtagname"])
def my_task():
pass
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.