Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Got a general question about configuring a flow to execute on AWS ECS EC2 instance instead of  ECS Fargate. Followed <https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c|this guide> to deploy everything using the latter but have discovered I need more resources than what’s <https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html|allowed> for Fargate task definitions. Is there an example/documentation for creating tasks that will execute on a custom EC2 instance?

Thanks much!

Hi <@U01KSKR589Z>! Does your process run on Dask?

<@U01QEJ9PP53>, no, at the moment it does not. It’s large read/transform operation that does involve loading a pandas data frame in memory 

Hey <@U01KSKR589Z>, you would need to make an EC2 instance and write your own task definition for ECS <https://docs.aws.amazon.com/AmazonECS/latest/developerguide/getting-started-ecs-ec2.html>

You can also just spin up an EC2 with the LocalAgent and run it like that.

Understood, that’s very helpful. Thank you!