https://prefect.io logo
Title
c

Colin Bieberstein

03/14/2023, 3:27 PM
I am attempting to use a flow to launch a number of parallel flows (up to 45) using RunDeployment against ECS Fargate (recursive flattening of json from a NOSQL db and expansion into child tables) . I note two big issues and wonder if anyone else has seen this. 1) ECS execution time is very slow, like clock speed is turned way down or something like that 2) Startup of new ECS containers takes ~2 minutes but you need to wait 30 seconds or so between container startup or you get throttled. We've tested giving additional CPU resources/RAM to the containers with no measurable impact. Each of the flows runs for between 5 minutes and 30 minutes depending on data change rates. I expect that there's something fundamentally wrong about the way we're deploying to Fargate using Prefect given this but can't see what it might be. Suggestions?
r

redsquare

03/14/2023, 3:34 PM
Your paying the well known Fargate penalty. A fresh node every time means having to pull down the image on every launch. no caching - prefect image is 400mb or so if your using vanilla
eks+ec2 is the way to go if you need speed
c

Colin Bieberstein

03/14/2023, 3:40 PM
Thanks, that explains the startup time, but not the slow execution time once started.
r

redsquare

03/14/2023, 3:44 PM
Fargate can put you on on any underlying metal that meets the CPU/RAM as per the task definition thus can have huge variance in perf - have you tried container insights to view any traits between slow/fast instances
are you pulling a large amount of data from rds/mongo or similar