Hi all! Maybe we have a weird use case, but I thought it should be possible with prefect : All our processes are dockerized, and we want to orchestrate them on AWS Fargate via an ECS Agent. We do not want to put the prefect flows on Fargate, but just instance tasks (e.g. running a container with a specific set of parameters). Now the conceptional question : what kind of task do I run? Do I need to also install prefect inside my docker image? Locally, I would just run my docker image via docker run or use the

docker.StartContainer

task. Thanks for your help!

Hey @Marcel Gutsche, I think you want the CreateContainer and StartContainer Prefect tasks. I don’t think you would need Prefect in these images since Prefect would just start them.

Hi @Kevin Kho, thanks for your reply! Is this also the case, when I use the ECS Agent for AWS fargate? As I have understood it, it will already run an image by giving a RunConfiguration. The docs on aws ecs are a little bit thin.

Ah I see what you are saying. The difficulty here is that Prefect registers the Flow as an ECS task with a container. I don’t think you can spin a container in ECS unfortunately. You might need to create it as an ECS task on your own, and then use Prefect to trigger to trigger that ECS task. Does that make sense?

How would such an ECS task look like? Do you mean that I have to run the main process inside docker by invoking it via a prefect task?

You register an ECS task through the AWS API (boto3) and then the ECSRun will literally be your container.

I'll give it a try! Thanks for the help and I will get back if I get stuck!

Oh didn’t have time to flesh out my thinking here earlier. So ECS as a service runs a Docker container already. If you use Prefect to run a Docker container, it becomes a weird Docker in Docker scenario, which is normally best to avoid. So the solution would be to register your container and whatever you want to do as an ECS Task already and then you’ll have a separate Prefect Flow that will just invoke that task

That's exactly what I wanted to do! Do I have to write it myself using boto3 or is there already an inbuilt "RunECS Task" available? We already have registered the ECS task, mangled all the permission, and now the last ingredient would be: RunECS Task with some parameters.

There is not RunECS task. You’d need to find a way to hit the API. Yes that is right. It can be a LocalAgent living on ECS also. I think we’d gladly accept a RunECS Task though if ever you build one 🙂 . We don’t have one at the moment

I accidently logged in with my private email, stupid me 😅