Hello Prefect team! Hope you all are doing well 🙂 We are currently using Prefect 1.0 to launch tasks on AWS ECS using a local agent installed on an EC2 machine. The reason we are using a local agent is because we plan to do the same on other clouds (namely Azure and GCP) as well, and since they dont have their specific native agents (like ECSAgent for ECS), the local agent seemed more universal/generic approach for a multi cloud environment. We had some questions regarding the above however: 1. Using the local agent with the Prefect Cloud, is there a hard limitation or some throttling/delays in starting flows that can occur if we schedule a lot of tasks concurrently. For example running 250 flows (and the tasks therein) concurrently. Can a local agent support this? 2. I noticed Prefect 2.0 coming up soon, however upon reading the docs there, things like ECSAgent etc do not seem to exist there, and I was wondering what the migration process from 1.0 to 2.0 would look like. Basically - we would like to run our flows in a multi cloud setup (1 agent per cloud, tagged with respective labels). Thanks!

Hi @Gaurav, The local agent may slow down if overloaded but it all depends on the available specs. If you need some kind of autoscaling, it might be worth it looking into Kubernetes. The Local agent is just like running Python scripts on a computer. They will run fine as long as there are resources. The thing to note though is if you have tasks that try to occupy all resources of the machine like some machine learning models. These will hit each other and you can’t really run them currently. Yes there is no ECSAgent but local, docker, and k8s flow runners are there. The migration should not take more than 2-3 hours and we’ll write the migration guide around it. For one agent per cloud, you’ll likely be doing Local/Dokcer/Kubernetes anyway, so actually you should already try Cloud 2.0

thanks for the prompt reply Kevin! Correct, given multi cloud our aim is for the flows itself to be lightweight, where the tasks are just going to itself be calls to spin up docker containers and wait for it to finish, rather than perform the computation of each task locally on the machine where the agent is installed.

For something lightweight it should be fine

@Gaurav, for ECS agent support in 2.0, check this post Regarding a migration guide, we started collecting various resources here. This discussion from today may also be helpful

Thank you Anna! Will certainly give that a read