h

    Hugo Kitano

    1 year ago
    Hello all, I am trying to reimagine an AWS Step Function, consisting of Lambda Functions and Fargate tasks, as a single flow. Lambda Functions can easily be reconfigured as Prefect tasks, but Fargate tasks, which require their own Docker containers, are a little bit more complicated. The
    ECSRun
    functionality could work, but it is designed for running flows, and this would necessitate flows in flows which could be a little tricky. What would be the best way to reconfigure a Fargate task as a Prefect task?
    nicholas

    nicholas

    1 year ago
    Hi @Hugo Kitano - one method that comes to mind is to use something like a Docker sidecar pattern, where you wrap/use the existing container orchestration tasks to orchestrate your previously fargate-based containers. Something like this
    h

    Hugo Kitano

    1 year ago
    Awesome, thanks Nicholas. I would be planning to run the docker sidecar off from an ECSRun, then. Some followup questions: 1. Do you know if it’s possible to pull the container off ECS instead of locally? 2. Looks like I wouldn’t have to worry about a custom Docker server URL if I were running on Fargate
    nicholas

    nicholas

    1 year ago
    1. You should be able to, using the PullImage task! 2. That's correct, I believe the docker tasks default to the normal docker endpoint (
    unix:///var/run/docker.sock
    )
    h

    Hugo Kitano

    1 year ago
    I see, so I can use my ECR URI as the repository parameter for that. Thanks!
    nicholas

    nicholas

    1 year ago
    Yup! You will likely need to add credentials to access that but it sounds like you've got that part under control 🙂
    h

    Hugo Kitano

    1 year ago
    @nicholas One last thing: will I need to be using Kubernetes to leverage this Docker sidecar functionality? Or is it possible to do this from an ECSRun?
    similarly, if I were to start a container within an ECSRun, wouldn’t that cause a Docker container within a Docker container? If so, not sure if that’s best
    nicholas

    nicholas

    1 year ago
    I don't think it's a requirement to use k8s or docker in docker for this - so long as you can provide an endpoint that allows deploying a container (like ECS), you shouldn't need to use either of those; the one caveat here is that this wouldn't be something that Prefect would orchestrate for you automatically; you'd provide the endpoint and define your containers as part of the CreateContainer task
    h

    Hugo Kitano

    1 year ago
    Yes, this sounds good: we may have multiple tasks running in parallel on different docker containers, so we’ll probably be deploying through ECS. Do you think setting off a Fargate task with boto3 as a task would work? Or are you proposing something different
    nicholas

    nicholas

    1 year ago
    Honestly that's probably fine as well - it's really up to preference. If you already have fargate tasks configured, it might just be easiest to stick with that for orchestration while still tracking them from your Prefect task
    You can turn on the boto3 logger and get all the logs you'd expect in Prefect
    h

    Hugo Kitano

    1 year ago
    awesome, that was probably my biggest concern with doing that (as well as waiting for the Fargate task to end, but I think boto3 can handle that). For the logging, I’ll probably want to do this, right? https://docs.prefect.io/core/concepts/logging.html#extra-loggers
    nicholas

    nicholas

    1 year ago
    Yup exactly!
    Prefect uses boto3 under the hood for most of the pre-built AWS tasks so if you're looking for inspiration those might be a good place to look