Hey, all! I'm Luis and work at Intuitive Care, a Brazilian healthtech. We are considering using prefect opensource and i was is there a reference implementation for prefect on AWS using ECS and isolating server, database and workers?

My team has cobbled together something close to what you’ve described on Prefect 1.0 and are generally pleased with how its been working. Not sure I would call it a reference architecture but roughly speaking we have • An elastic beanstalk (👎 ) service which deploys a modified version of the docker compose file for prefect server. Most of the modification involve pointing the services at an external DB instance in rds. Generally have been happy with this but would like to migrate it to ECS (as its own distinct service) • A long running ECS task for the agent. Its mostly just the

prefecthq/prefect:1.2.3-python3.7

image with some extra config files for host ips, subnets, and ECS settings for the flow runs. • A collection of prefect flows, wrapped in a docker file which is deployed to ECS. This includes a set of common python libs + utils. All flow runs go to the single ECS agent and are run using an ECSRun config (pointing to the image from the flows repo). It does not support multiple agents as our traffic is quite low, but I dont see a reason why it could not (just deploy multiple agent tasks in ECS and use labels). I would be happy to answer more detailed questions in depth if you have them (might be a bit async as I work on other things)

Thanks, @Kyle McChesney! I was estimating the number of tasks to be executed daily here, we estimated about 4 million/day, did your environment handle about this volume? Was it much less or much more?

The database seems to be managed by the deployment of prefect server via compose. I dont know the specifics here, but we never touched the tables/indices and they are present. As for scaling, our volume is MUCH lower. We use it for some in house biological data analysis that our lab teams use. To date (~1 year) we have only run 21k tasks IN TOTAL. I cannot comment much on where the anticipated issues would be with scaling up this large. Using ECS should work well enough for scaling your agents (and flow runs if you go that way) since you are just using AWS scaling for that. I’d imagine the server deployment, database instance, etc would need to be scaled up. Either big servers or horizontally scaled. Again, I dont have much insight on how that looks.

Ok, Thanks!

also if you do get this up and running and scale up that large, I would be VERY interested in hearing about it!