Hey Prefect - I have one agent and one queue, I ran a loop over some parameters to break up some work and called

run_deployment

to essentially launch 30 concurrent ecs tasks. What I notice is that 5 of the deployment runs start and the other 25 are pending. How do I make this behave similar to prefect 1.0 where I was able to have all of these run at the same time ? Is this a limitation of my agent ? I was under the impression all the agent does is orchestrate the ecs tasks in my aws environment?

can you share your code example? we can try to reproduce

this looks like a bug - it shouldn't need idempotency key set manually

I dont have a local agent

?

ok thanks. I was just saying dam because I have to run it in order lol

nice! I tested it just now locally and works fine

so it is a ECS thing? I still think it is a bug

probably a smart choice to put some time in between to avoid rate limiting issues with AWS API and serverless provisioning madness, but all of those scheduled and pending runs got executed in the end and they should

yeah interested to see what you get

ok so here's my take - I don't think this is something that we should solve on the Prefect end - the only problem I see is still registering so many revisions of the same task definitions family concurrently which honestly makes sense if you are running things at scale, you should register task definition once and then trigger a bunch of containers from the same task deifnition

ok that is fair enough - why was this not an issue with prefect 1.0 though? isnt that a step backwards?

so my recommended approach for the solution is either: 1. add some sleep time in between as you suggested, this will solve it while still keeping this build step during runtime 2. register task definition once at build time e.g. from CI and reuse that, i.e. provide task definition arn on the block based on what you said earlier, #1 seems more convenient/applicable for your use case

yeah cool - will do. thanks

thanks for flagging, we definitely need to keep testing edge cases and do more of such scale testing, thanks so much for the positive push here 🙌

Thanks for the quick reply and help

I think we can figure out one day when we find the time some middle ground with task definition caching so that we keep reregistering when needed and skip that otherwise to keep the convenience without ECS throttling

Ah great! I will have a read after my lunch 🙂

it's just a summary, nothing new

Hi @Anna Geller there is actually more to this story and it is quite concerning tbh. When I have multiple deployments that happen to be scheduled at the same time eg

0 * * * *

there appears like maybe there is a race condition with the agent/queue and the flows that dont get in first, stay in the

Pending

state indefinitely. I just logged onto my UI and I have flows that are up to 3 hours late and just never get started.

I couldn't notice any issue for a Process run, could be some Fargate weirdness to explore. in that case, I would so much appreciate it if you could open an issue on the prefect-aws repo if this only affects ECSTask runs

I wouldn't know what exactly to write. I have messaged my account manager and probably best to start off there? Once I have a clearer idea I can do that

AE won't help you solve an infra problem, it would help if you would describe the problem the same you would to AE but on a more technical level in a GH issue. you know how fast Michael ships fixes, worth opening an issue for that reason alone 😜

yeah ok

(Michael wrote that infra block)