Thread
#prefect-community
    j

    Jamie McDonald

    8 months ago
    Is there an elegant way to control the level of concurrency using the
    map()
    functionality? My scenario is a list of URLs that should be used for making requests to but want to perform the requests in batches of 'n' rather than overwhelming a server with them all at once.
    Anna Geller

    Anna Geller

    8 months ago
    Yes, there is - you can leverage tags on your task decorator to set the tag, and then in the UI you can leverage task run concurrency to limit the number of concurrent task runs for that tag.
    from prefect import task
    
    @task(tags=["yourtagname"])
    def my_task():
        pass
    j

    Jamie McDonald

    8 months ago
    Perfect, thanks @Anna Geller