hi, I just moved from prefect1 to prefect2 and having some issues with running agents n flows in ecs.
I have flow that deploys another flow at scale (using same task definition, no reregister).
the flows are created in the ui but
• some of them(quite a few) are stuck in pending n not picked by the agent
• I think agent is crashing because all of the logs being pulled there (can I disable agent pulling in logs? I assumed stream_output in infrastructure was meant to do that, but that didn't help much)
I also noticed that at a time, only around 40 ecs tasks are being created at a time, n the next set of tasks are created only after these ones are completed.
didn't have this issue in prefect1, could easily run around 500 tasks with no issue
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.