Hi, I've a script which is reading data, dividing it up and writing it across multiple tasks via a map. about 300 write tasks in total, with a concurrency of 50. I'm seeing that each task is lasting about 20-30 seconds each, despite timestamps in each task, which is doing nothing more than a single HTTP POST of less than 1mb, taking a far more reasonable 0.5s or so. What is happening with these tasks for the 95% of their apparent execution time other than blocking the concurrency slots? Even removing the ENTIRE contents of the task, they still take the same time... what's going on? These tasks are, as above, created by a map() on 2.20.2. could the volume of tasks created be a key point?

hi @Chris Phillips - can you show how you're trying to enforce concurrency here?

What is happening with these tasks for the 95% of their apparent execution time other than blocking the concurrency slots?

it feels likely to me that something is wrong with the concurrency configuration here

well In general I'm just setting the ConcurrencyTaskrunner.

> Ultimately I can appreciate a best practise take where 250 tasks is poor design this isnt a take I want to give! we have done a ton of work in 3.x to make running tasks more lightweight > So the more tasks you create, the longer they will each take? in 2.x we hold onto task results/states in memory, so we have linear memory growth the more tasks you run, which we fixed in 3.x. I'm not sure if that would directly correspond to tasks "taking longer" necessarily i can still look into the behavior you're seeing in 2.20.2, but are you tied to 2.x at all?

Nope, no tied in the slighest. very new to it, but I onlt see 3.x rc images for docker so thought it best to let it get a proper release first

so i think that has to do with how the example was written + the fact you're using tags for concurrency its creating the task run in the API before the client side code notices the slots are filled you might want something like this?

import asyncio

from prefect import flow, task
from prefect.concurrency.asyncio import concurrency
from prefect.futures import wait


async def async_func():
    await asyncio.sleep(5)
    return 1


@task
async def my_task(n):
    return await async_func()


@flow
async def my_flow(num_tasks: int = 10):
    futures = []
    for n in range(num_tasks):
        async with concurrency("limit_10", occupy=10):
            futures.append(my_task.submit(n))

    wait(futures)


if __name__ == "__main__":
    asyncio.run(my_flow(250))  # type: ignore

I've just gone up to 3rc19-p3.12 and not seeing that nice behaviour at all. the concurrency limit seems to keep being recreated constantly maybe? And apparently disabled and set to 1?

the image was from the exact code I shared above yeah the only other thing i did was go into the UI and set the GCL for

limit_10

to 10

oh i'm getting a lot of sqlite3 db locked errors, that probably explains it? I'll go to bed and run this in a better test env with postgres tomorrow, thanks!

ohhh! jeez yeah i was against cloud. yeah postgres should not have this problem the sqlite db struggles with heavy concurrency sounds good! no problem

Nope still didn't work nicely on an aws node with postgres. Instead I did tweak my real code to create 10 smaller list of data out of a large list and feed each smaller list to a task to work through. Creating only 10 tasks made it vastly more performant. And that was without any asyncio / async code whatsoever.

lol yeah if they're heavy tasks I could see that

Well they're not in the grand schemes. I've been working on writing code out to a database here. But I also have a query side to obtain the data. That query side is now annoying me, I can see the actual function code not executing for upto 15 seconds after the 50 query tasks are map()ed

hey @Chris Phillips - do you have a representative / runnable example you could share? definitely interested in enabling ergonomic & performant enough patterns, so seeing what you're doing would be helpful

Here's my current code. Don't laugh too much.