https://prefect.io logo
m

Mattias Petter Johansson

09/13/2023, 1:38 PM
@Marvin hey marvin! I now have used the https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent guide and after running it, I have configured an ecstask using prefect_aws that points to a docker image with dependencies needed for my flows to run. What I now don't get is that there is BOTH a concept of a ecs work pool and and ecs task block - and they seem to have overlapping configuration options. How do they interrelate?
m

Marvin

09/13/2023, 1:39 PM
Ah, the joys of AWS terminology. It's almost as if they designed it to be as confusing as possible. But let's wade through the mire together, shall we? An ECS Task is not the same thing as a Prefect task. ECS tasks are run as part of an ECS Cluster, and they launch container(s) as defined in the ECS Task definition. The ECS Task definition is essentially a blueprint for the ECS task that describes which Docker container(s) to run and what you want to have happen inside these container(s). Now, the ECS task running the Prefect worker should be set up as an ECS service. Why, you ask? Well, because it's a long-running process, and you want it to be rebuilt automatically if it ever stops unexpectedly. ECS services are used to guarantee that you always have some number of Tasks running at all times. For example, if a Task’s container exits due to an error, or the underlying EC2 instance fails and is replaced, the ECS Service will replace the failed Task. This makes ECS services perfect for managing a long-running process like the Prefect Worker. On the other hand, ECS Tasks are instances of a Task Definition. An ECS Task Execution (as opposed to service execution) launches container(s) (as defined in the task definition) until they are stopped or exit on their own. This makes ECS Task executions perfect for running jobs that have a definite start and end. So, in essence, an ECS service is for long-running tasks that need to be always on, while an ECS task is for jobs that run until completion. I hope that clears up the confusion. If not, well, I tried. It's not like I'm a human with an innate understanding of these things. Source