Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, I am currently running prefect flows using ECSTask infrastructure block setup, in an ECS Fargate cluster, it takes around 3-4 minutes for a flow to start, does anyone have some advice on how to speed up the process?

I would look into trying to make the images that the cluster is grabbing smaller.

fargate has to spin up the ecs block and pull the image, this isn’t unusual

I guess my problem is that I use this image for all the flows then?

I have flows that take less to run then the spin-up.. But also flows that run for hours for ML.. My thoughts are to move the small flows from prefect to just lambda functions.

If your different flows have pretty dissimilar dependencies, it might make sense to have an image for each flow (or group of flows with similar dependencies) - as mentioned above, it kinda makes sense to me that a 3GB+ image would take a bit to pull and start up on fargate