https://prefect.io logo
a

Andrei Tulbure

10/15/2022, 3:11 PM
Hello all. Any idea why the Prefect Flows are freezing when running them in ECS with Prefect Cloud ? They freeze and in the cloud they appear as they are not finished, but they never actually do anything after a certain period. And the ECS machines I use are the largest one possible. ?
a

Anna Geller

10/15/2022, 6:54 PM
Not enough information to say atm. Could you explain your setup, the Prefect version you used and the steps you took? Alternatively, do you want to try out this repo template? README has a blog post and video explanation https://github.com/anna-geller/dataflow-ops
a

Andrei Tulbure

10/18/2022, 5:25 PM
@Anna Geller I am using Prefect 1. And the old 1.x Prefect Cloud. The setup is Prefect flows are registered on AWS ECS with Fargate on 16vcpu machines. When I run 4-6 flows concurrently, there is no issue. Once I pass 7-9 concurrent flows, some of them freeze. They still appear as alive but when i check the logs they are dead. The machines are big enough to run them, the flows usually take abt 8-10 hrs at most and my aws account has enough vCpus available for fargate to run them
@Anna Geller today it happened again and it is sadly annoying
a

Anna Geller

10/18/2022, 5:31 PM
the freezes may occur due to rate limits when registering too many ECS task definitions at once, to solve that you may try registering your task definition yourself and pass ARN explicitly this can also be a good motivation to start migrating over to Prefect 2 where this should no longer be an issue
a

Andrei Tulbure

10/18/2022, 6:43 PM
Thanks! Will definitely look into it
👍 1
3 Views