example, I have a task that submits about 200 subtasks. You can see there is about 1 minute 30 seconds latency before any of them start

can you show what your code looks like? it likely has to do with the tasks you're submitting

In [1]: from prefect import flow, task

In [2]: @flow
   ...: def f(): task(lambda x: x +1).map(range(200)).result()

In [3]: f()

since this takes about 3 seconds to finish in prefect 3.x

ah you might want as_completed here? https://github.com/prefecthq/prefect/blob/main/src/prefect/futures.py#L333

can you elaborate please?

still not 100% sure I understand. certainly the lines I linked are blocking. But the task that is waiting on them is itself submitted as an aysnc task using

task.submit

https://github.com/MCGallaspy/pokemon_showdown_replay_tools/blob/main/scripts/populate.py#L161C9-L161C22 yeah my assumption in looking at the code was that this

future.wait()

was blocking the main thread for each future, I could be wrong

ok, I think I understand. ty for the help and the great gist

👍 feel free to post back here if you're still seeing delays

is this blocking? https://gist.github.com/zzstoatzz/2f36e7c64009e5e391510794c03e2d92#file-pokemon_replay-py-L193-L197

hrmmmmm yes. but once a future comes out of

as_completed

you know its done so the

future.result()

call which would normally be blocking should happen instantly but overall yeah the list comp would be blocking bc I'm exhausting

as_completed

instead of doing something

for

each completed future that pops out

I'm just trying to understand it myself, I don't have a minimal example

maybe

I ran your script without modification and I'm still seeing big latency. I'm assuming it's a me issue at this point. I wonder if there's any way to profile the prefect internals?

hrm what does

prefect config view

say? i.e. are you running against an ephemeral server, oss server, or cloud?

looks like an open source server. so you have

prefect server start

going someplace?

yup

hm. and the script you copied w/o modification, thats the gist I shared?

yeah

huh - off the top of my head im not sure is the delay only in the resulting timeline in the UI or do you see the work literally delayed in your terminal?

it's literally delayed

im tempted to point you here (a relatively new pattern in prefect), but im not sure how literally you mean consumer/producer what im sharing above is akin to redis streams / celery, where you have • finite set of tasks you serve (this is a websocket client that gets pushed task runs from the server) i.e.

serve(*many_tasks)

◦ you can horizontally scale these (e.g. N pods) arbitrarily without race conditions because of consumer groups ◦ all the task features apply, i.e caching, results, retries etc • from somewhere like a webapp, you can

some_task.delay(**task_kwargs)

to "background" that task without blocking so this is good for cases where you want to offload a bunch of work to happen concurrently somewhere on static infra, but the caller (or delay-er) doesn't need the result of that background task am I going off the rails here or does that sound like something you're interested in?

you're not going off the rails

what a metaphorical consumer/producer pattern is

haha fair enough. well cool, those examples are almost all docker-compose and should be mostly up to date, lmk if you have any specific questions