Hi everyone - i’m having some issues of concurrenc...
# ask-community
g
Hi everyone - i’m having some issues of concurrency of task and parallelization here. I’ve been around the entire documentation and implemented all the ways i saw possible and i can’t make them work on my end. Current prefect version: 2.14.5 I’m gonna resume a little bit of what i’m trying to achieve: Limit task concurrency (to 400) while running the task in parallel with Dask i’m currently trying to run a map in one of the task created:
Copy code
@flow(name='pe_api_client_agreements_flow'
    ,task_runner=DaskTaskRunner())
    
def flow_run():    
    list_id = get_list_id()
    returns = get_revisions_rate_limit.map(list_id)
I’ve try multiple ways on trying to limit the concurrency: 1. Task concurrency tags (@task(tags=[‘prefect_api_concurrency_limit’]) + Agent pool(Task Run Concurrency Limits) a. Created and submitted tasks just passed by way over the limitation - Doesn’t work 2. Global concurrency limits (UI + code) - Limits are activated on the UI a. It doesn’t limit the the submitted task created - Doesn’t work It’s not well defined what needs to be done in what level and if should be done both on UI and code. I’ve tested a bunch of test case scenarios and don’t see any of the provided tools working here. Anyone have thoughts on this? Any help is more than appreciated!
👀 1
Here does show up that the concurrency is “activated” but doesn’t work on limiting the concurrency
y
Following. have the same issue