https://prefect.io logo
#prefect-server
Title
# prefect-server
m

Marcel Gutsche

09/10/2021, 9:32 AM
Hi all! Maybe we have a weird use case, but I thought it should be possible with prefect : All our processes are dockerized, and we want to orchestrate them on AWS Fargate via an ECS Agent. We do not want to put the prefect flows on Fargate, but just instance tasks (e.g. running a container with a specific set of parameters). Now the conceptional question : what kind of task do I run? Do I need to also install prefect inside my docker image? Locally, I would just run my docker image via docker run or use the
docker.StartContainer
task. Thanks for your help!
k

Kevin Kho

09/10/2021, 2:22 PM
Hey @Marcel Gutsche, I think you want the CreateContainer and StartContainer Prefect tasks. I don’t think you would need Prefect in these images since Prefect would just start them.
m

Marcel Gutsche

09/10/2021, 4:40 PM
Hi @Kevin Kho, thanks for your reply! Is this also the case, when I use the ECS Agent for AWS fargate? As I have understood it, it will already run an image by giving a RunConfiguration. The docs on aws ecs are a little bit thin.
k

Kevin Kho

09/10/2021, 4:45 PM
Ah I see what you are saying. The difficulty here is that Prefect registers the Flow as an ECS task with a container. I don’t think you can spin a container in ECS unfortunately. You might need to create it as an ECS task on your own, and then use Prefect to trigger to trigger that ECS task. Does that make sense?
m

Marcel Gutsche

09/10/2021, 4:50 PM
How would such an ECS task look like? Do you mean that I have to run the main process inside docker by invoking it via a prefect task?
k

Kevin Kho

09/10/2021, 4:52 PM
You register an ECS task through the AWS API (boto3) and then the ECSRun will literally be your container.
m

Marcel Gutsche

09/10/2021, 4:55 PM
I'll give it a try! Thanks for the help and I will get back if I get stuck!
k

Kevin Kho

09/10/2021, 9:08 PM
Oh didn’t have time to flesh out my thinking here earlier. So ECS as a service runs a Docker container already. If you use Prefect to run a Docker container, it becomes a weird Docker in Docker scenario, which is normally best to avoid. So the solution would be to register your container and whatever you want to do as an ECS Task already and then you’ll have a separate Prefect Flow that will just invoke that task
👍 1
m

Marcel Gutsche

09/13/2021, 8:22 AM
That's exactly what I wanted to do! Do I have to write it myself using boto3 or is there already an inbuilt "RunECS Task" available? We already have registered the ECS task, mangled all the permission, and now the last ingredient would be: RunECS Task with some parameters.
And one more question related to this: I guess I will not need an ECSAgent, but a local Agent, because the Agent does not sit on Fargate/ECS but just spawns tasks?
k

Kevin Kho

09/13/2021, 3:49 PM
There is not RunECS task. You’d need to find a way to hit the API. Yes that is right. It can be a LocalAgent living on ECS also. I think we’d gladly accept a RunECS Task though if ever you build one 🙂 . We don’t have one at the moment
You have two Slack accounts? lol
m

Marcel Gutsche

09/14/2021, 11:52 AM
I accidently logged in with my private email, stupid me 😅
2 Views