Quick question. I'm going to be building a complex data processing workflow in AWS and we plan to use our own Prefect installation for this. I'm trying to determine if it should use AWS EKS (kubernetes) or AWS ECS (elastic container service). We plan to dockerize most of our worker nodes and call those from Prefect (they'll be running on windows and need custom installed software often, not just raw python, so we need to use EC2 instances not Fargate). Would anyone have a recommendation for going with EKS or ECS for a use case like this? My assumption is that there will be more community support when using Prefect with EKS as that's a more commonly used open source solution? Just want to go with whatever makes our lives the easiest.

I’d actually question wether you need either based on your description… just install docker on your windows ec2 instances and put them in an autoscaling group based on your load (mem, disk io, cpu)

It's not that kind of project. We will need 20+ individual worker services with different docker images in separate scaling groups. Additionally we cannot scale them based on mem, disk io or CPU, as most of the time we need 0 instances, but based on an employee scheduling a job we'll need an arbitrary number of worker nodes spun up to accomplish each step in a specific task (and to parallelize within an individual task). I was originally planning on creating an SQS queue so we could scale the individual workers in different services based on the contents of the SQS queue, but it's my understanding that I could have prefect programmatically scale workers for an individual task in a Prefect workflow?

hi Nova, just found this thread by searching for various AWS pieces I'm probably going to use for my project too. I dunno if you already found this terraform example to run an agent on ECS Fargate by the Prefect team. It worked with basically no changes on my part

Unfortunately I'm not actually further along! I was working more on a frontend project to go with the project / a presentation. I'm just starting to get going on the Prefect part of the project. I did see recently that they released some more tools for Prefect + ECS tho.

ahh gotcha. yeah I'm thinking about Prefect as a way to manage work dependencies, retries, and state-of-work. like there are probably a few DB tables I won't have to come up with if I use an orchestration tool but the way I'm going to run it is 1. some ECS task(s) running Prefect, waiting for work. they kick off AWS batch jobs (which have much higher compute/mem requirements) 2. then when the batch jobs have finished (and I somehow know that), ... do something else in the workflow, I guess

Certainly there's a few DB tables and some code you don't need to write if you use Prefect! Is the pay off worth it for my specific use case tho? TBD. 😄 But yeah that is a bit confusing to me if there's no built in way for Prefect to detect asynchronously the completion of arbitrary tasks in a flow to then continue running the flow? Is that sort of the most basic level of necessary things to make the system work with remote processes / workers?

yeah... I feel like if I could use ECS tasks or call Lambda workers or something that I can just sit with an open socket and wait for a response from, that'd be a use case that fits Prefect better. but AWS Batch doesn't work like that, and is probably a requirement for me from a computing load pov

Yeah, so long as they can be done with asynchronous connections in that regard, for sure. I haven't used AWS Batch / don't know much about it.