Braun Reyes

    Braun Reyes

    1 year ago
    Hello yall! Long time no contribute....we were wanted to make a couple enhancements to the fargate agent. 1 - add an
    extra_containers
    argument to support adding sidecars to the flow run container 2 - add the ability to skip the task definition registration based a run context key of
    task_arn_override
    . This would allow folks using a map to submit flow run(subflow pattern) to skip the task registration issue that @Darragh and @Lukas were running into
    Darragh

    Darragh

    1 year ago
    Number one would actually be super useful for us at the moment - we need to have a side car container run beside the flow container that we cna mount volumes in and out of, so extra containers for fargate tasks would be a godsend 😁🙏
    And 2 as well obviously 👍
    Braun Reyes

    Braun Reyes

    1 year ago
    cool
    Darragh

    Darragh

    1 year ago
    Definitely a beer in it if you pull it off 😁
    j

    josh

    1 year ago
    Yeah @Braun Reyes those sound like great enhancements to the Fargate agent! @Jim Crist-Harif is actually currently working on a new experimental feature called Run Configs (which is somewhat comparable to Environments) and in the process of building an ECSRunConfig and a new ECSAgent (ECS to support both Fargate and EC2). This is effectively moving the configuration to the flows and off of the agent. This will allow for things like cleaner configuration, changing configuration after registration, removing the need for the initial prefect task (that currently loads the evironment), etc. I’ll let Jim add in to this if he wants but otherwise more details coming soon 🙂 Note: the Fargate Agent isn’t going anywhere and it will stick around to support the current Environment pattern!
    Braun Reyes

    Braun Reyes

    1 year ago
    awesome to hear...will be super cool to see the evolution
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    PR should hopefully be up sometime early this week. The short story is flows can be configured with: • a task definition name (using aws's terminology, this is either
    family
    ,
    family:revision
    , or full task def arn). If not provided, the default task definition name set on the agent will be used. • Any "overrides" for that task definition to set at task startup. This includes support for basically anything you can pass to
    run_task
    (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task). Some common parameters like
    image
    /cpu /memory /etc... have been pulled out for easy access, but users can configure anything to pass to
    run_task
    on the flow itself. The ECS agent is then configured with a default task definition to use (if not provided, it will auto-register a basic definition that ships with prefect). No other task definitions will be auto-registered by prefect, if you want to provide new custom definitions you'd need to register those yourself. To simplify registering custom task definitions for users, we'll also provide a helper in the prefect cli as
    prefect agent ecs register-task-definitions ...
    . This means that no definitions are registered at agent runtime, all the agent does is all
    run_task
    with an already existing definition and a set of configurable parameters (either configurable on the flow or the agent, with the flow taking precedence). The hope is that this design can support everything a user/team might want, while also being much simpler to maintain and reason about.
    Braun Reyes

    Braun Reyes

    1 year ago
    Yeah this is much better long term...do you think we should continue down this path then? Seems like this new agent satisfies extra containers and task definition arn overrides
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Up to you. Run configs are a breaking change - the new
    ECSAgent
    will only support flows configured with a
    flow.run_config
    (not a
    flow.environment
    ). Depending on how fast you think you can transition over to the new model, it might still be worth your time to patch the old fargate agent.
    We're going to have a loooooong deprecation cycle for both environment-based configuration, and the fargate agent. Can't ask people to move the world that quickly.
    Braun Reyes

    Braun Reyes

    1 year ago
    makes total sense
    thank you...we will push forward then
    oh and @Jim Crist-Harif I appreciate your wood working skills on twitter
    l

    Lukas

    1 year ago
    This sounds amazing, @Jim Crist-Harif so we'll be able to configure memory / cpu per flow, do I understand it correctly?
    j

    josh

    1 year ago
    @Lukas yes!
    Darragh

    Darragh

    1 year ago
    Is there a PR for it guys (@Jim Crist-Harif @josh), just to read read up on? Interested to see how it would compare against the current agent and env setup
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    As I said above, not yet. Should be up sometime early this week.
    Darragh

    Darragh

    1 year ago
    If context for our use case would be any use, we need to run our main flow container in a fargate task, with a second specialised side car container that needs to be isolated for security (no common env between the two, only specific and controlled volumes mounted, etc...) . It sounds like Braun's change will help us solve this, so I'm very happy there 😁 so I'd love to eyeball the requirements for the config works see what's involved there . Apologies Jim, PR was the wrong phrase. Requirement, pin , request, whichever is the correct term
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Sorry, most of the design discussions for this have been internal. The end goal is simpler configuration of flow runs for all backends. Kubernetes, docker, and local runs are already completed, ECS is the last remaining one. Given your described use case above, the new design should handle that fine. An admin/permissioned user would register a task definition to use that contained the extra container description. Flows that wanted to use that definition would pass it as
    task_definition
    to
    ECSRun
    , or if all flows should use it an admin would configure the ecs agent to use it as the default definition.
    I'll be sure to reply to this conversation once the PR is up with a link
    Braun Reyes

    Braun Reyes

    1 year ago
    @Mark McDonald
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Example user-side config:
    from prefect.run_configs import ECSRun
    
    # Use the defaults set on the agent
    flow.run_config = ECSRun()
    
    # Use the default task definition on the agent, but override cpu and the image
    flow.run_config = ECSRun(image="my-fancy-image", cpu="2 vcpu")
    
    # Use an alternate task definition
    flow.run_config = ECSRun(task_definition="my-task-definition")
    
    # Override some parameters to run_task
    flow.run_config = ECSRun(run_task_kwargs={"overrides": {"executionRoleArn": "..."}})
    Darragh

    Darragh

    1 year ago
    Great, thanks Jim! You mentioned permissions user, is that coming to Core?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Yeah. By "permissioned" I just mean users that have permission to register task definitions in AWS (which might not be everyone in some teams).
    Darragh

    Darragh

    1 year ago
    Ah ok, got it. I thought you meant users in the UI 😁 thanks for all the info, looking forward to seeing the changes!
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Here's the PR: https://github.com/PrefectHQ/prefect/pull/3585 Note that things don't work quite the way I explained them above (turns out you can't override
    image
    at runtime, which prefect would require for that model to work). The top comment in the PR should explain how things work as currently written. If any of y'all have feedback on this design, I'd be glad to hear it. The goal here is to make configuring and running prefect on ECS much easier.. cc @Darragh, @Braun Reyes, @Lukas
    Besides general feedback, I have some specific UX questions listed here: https://github.com/PrefectHQ/prefect/pull/3585#issuecomment-718258787
    Darragh

    Darragh

    1 year ago
    Thanks for getting back to us with that @Jim Crist-Harif, much appreciated! Having a read now
    So if I read the PR right, in my case where I want multiple containers in a task_definition then I should be able to provide that as something like this…
    ECSRun(task_definition={containers: [{image: my_first_container}, {image: my_second_container, mount_volume: my_first_container/SOME_PATH]})
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Yeah, that'd work. Or if you want most flows to use that pattern, you could configure that on the agent as agent-side defaults.
    Darragh

    Darragh

    1 year ago
    Tried that before on the agent with an older version [0.12.6 I think??] and I could never get it to work. The flow specific one would work better, as we’d be able to configure resources specific to the side car container we’re running 👍
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Hopefully the new design makes it easier to configure agent-side defaults, if you ever feel the need to try that pattern. Flow-side configuration is more flexible though, and should handle what you need perfectly.
    This has since been merged, and will be part of the next release. If you're feeling bold (and want to try something with no documentation beyond docstrings like https://docs.prefect.io/api/latest/run_configs.html#ecsrun), please let me know how things work for you. Docs should be up in the next couple weeks.
    Darragh

    Darragh

    1 year ago
    Thanks Jim, much appreciated! I’ll try carve some time out tomorrow to test it, see if I can make it dance to my tune 😄 Rough idea when the next release is that this PR would be in?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Thanks Darragh. We're off today (election day), but we'll probably issue a release tomorrow.
    Braun Reyes

    Braun Reyes

    1 year ago
    its been along time since I posted on these threads where I was going to make some changes to make it possible to define a task definition outside of flow registration so that agent does not always attempt to register one.  There was also a change we were going to make to allow for sidecars as well.  Good news is that the new ECSRun and ECSAgent features handles this for you.  Here is a snippet from our deployment process.
    ECSRun(
            task_definition_arn=task_definition_arn, run_task_kwargs=dex_run_task_kwargs
        )
    Braun Reyes  [11:19 AM] the task definition can have the sidecar defined in it