Hello all, I am trying to reimagine an AWS Step Function, consisting of Lambda Functions and Fargate tasks, as a single flow. Lambda Functions can easily be reconfigured as Prefect tasks, but Fargate tasks, which require their own Docker containers, are a little bit more complicated. The

ECSRun

functionality could work, but it is designed for running flows, and this would necessitate flows in flows which could be a little tricky. What would be the best way to reconfigure a Fargate task as a Prefect task?

Hi @Hugo Kitano - one method that comes to mind is to use something like a Docker sidecar pattern, where you wrap/use the existing container orchestration tasks to orchestrate your previously fargate-based containers. Something like this

Awesome, thanks Nicholas. I would be planning to run the docker sidecar off from an ECSRun, then. Some followup questions: 1. Do you know if it’s possible to pull the container off ECS instead of locally? 2. Looks like I wouldn’t have to worry about a custom Docker server URL if I were running on Fargate

1. You should be able to, using the PullImage task! 2. That's correct, I believe the docker tasks default to the normal docker endpoint (

unix:///var/run/docker.sock

)

I see, so I can use my ECR URI as the repository parameter for that. Thanks!

Yup! You will likely need to add credentials to access that but it sounds like you've got that part under control 🙂

@nicholas One last thing: will I need to be using Kubernetes to leverage this Docker sidecar functionality? Or is it possible to do this from an ECSRun?

I don't think it's a requirement to use k8s or docker in docker for this - so long as you can provide an endpoint that allows deploying a container (like ECS), you shouldn't need to use either of those; the one caveat here is that this wouldn't be something that Prefect would orchestrate for you automatically; you'd provide the endpoint and define your containers as part of the CreateContainer task

Yes, this sounds good: we may have multiple tasks running in parallel on different docker containers, so we’ll probably be deploying through ECS. Do you think setting off a Fargate task with boto3 as a task would work? Or are you proposing something different

Honestly that's probably fine as well - it's really up to preference. If you already have fargate tasks configured, it might just be easiest to stick with that for orchestration while still tracking them from your Prefect task

awesome, that was probably my biggest concern with doing that (as well as waiting for the Fargate task to end, but I think boto3 can handle that). For the logging, I’ll probably want to do this, right? https://docs.prefect.io/core/concepts/logging.html#extra-loggers

Yup exactly!