Hello yall! Long time no contribute....we were wanted to make a couple enhancements to the fargate agent. 1 - add an

extra_containers

argument to support adding sidecars to the flow run container 2 - add the ability to skip the task definition registration based a run context key of

task_arn_override

. This would allow folks using a map to submit flow run(subflow pattern) to skip the task registration issue that @Darragh and @Lukas were running into

Number one would actually be super useful for us at the moment - we need to have a side car container run beside the flow container that we cna mount volumes in and out of, so extra containers for fargate tasks would be a godsend 😁🙏

cool

Definitely a beer in it if you pull it off 😁

Yeah @Braun Reyes those sound like great enhancements to the Fargate agent! @Jim Crist-Harif is actually currently working on a new experimental feature called Run Configs (which is somewhat comparable to Environments) and in the process of building an ECSRunConfig and a new ECSAgent (ECS to support both Fargate and EC2). This is effectively moving the configuration to the flows and off of the agent. This will allow for things like cleaner configuration, changing configuration after registration, removing the need for the initial prefect task (that currently loads the evironment), etc. I’ll let Jim add in to this if he wants but otherwise more details coming soon 🙂 Note: the Fargate Agent isn’t going anywhere and it will stick around to support the current Environment pattern!

awesome to hear...will be super cool to see the evolution

PR should hopefully be up sometime early this week. The short story is flows can be configured with: • a task definition name (using aws's terminology, this is either

family

,

family:revision

, or full task def arn). If not provided, the default task definition name set on the agent will be used. • Any "overrides" for that task definition to set at task startup. This includes support for basically anything you can pass to

run_task

(https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task). Some common parameters like

image

/`cpu` /`memory` /etc... have been pulled out for easy access, but users can configure anything to pass to

run_task

on the flow itself. The ECS agent is then configured with a default task definition to use (if not provided, it will auto-register a basic definition that ships with prefect). No other task definitions will be auto-registered by prefect, if you want to provide new custom definitions you'd need to register those yourself. To simplify registering custom task definitions for users, we'll also provide a helper in the prefect cli as

prefect agent ecs register-task-definitions ...

. This means that no definitions are registered at agent runtime, all the agent does is all

run_task

with an already existing definition and a set of configurable parameters (either configurable on the flow or the agent, with the flow taking precedence). The hope is that this design can support everything a user/team might want, while also being much simpler to maintain and reason about.

Yeah this is much better long term...do you think we should continue down this path then? Seems like this new agent satisfies extra containers and task definition arn overrides

Up to you. Run configs are a breaking change - the new

ECSAgent

will only support flows configured with a

flow.run_config

(not a

flow.environment

). Depending on how fast you think you can transition over to the new model, it might still be worth your time to patch the old fargate agent.

makes total sense

This sounds amazing, @Jim Crist-Harif so we'll be able to configure memory / cpu per flow, do I understand it correctly?

@Lukas yes!

Is there a PR for it guys (@Jim Crist-Harif @josh), just to read read up on? Interested to see how it would compare against the current agent and env setup

As I said above, not yet. Should be up sometime early this week.

If context for our use case would be any use, we need to run our main flow container in a fargate task, with a second specialised side car container that needs to be isolated for security (no common env between the two, only specific and controlled volumes mounted, etc...) . It sounds like Braun's change will help us solve this, so I'm very happy there 😁 so I'd love to eyeball the requirements for the config works see what's involved there . Apologies Jim, PR was the wrong phrase. Requirement, pin , request, whichever is the correct term

Sorry, most of the design discussions for this have been internal. The end goal is simpler configuration of flow runs for all backends. Kubernetes, docker, and local runs are already completed, ECS is the last remaining one. Given your described use case above, the new design should handle that fine. An admin/permissioned user would register a task definition to use that contained the extra container description. Flows that wanted to use that definition would pass it as

task_definition

to

ECSRun

, or if all flows should use it an admin would configure the ecs agent to use it as the default definition.

@Mark McDonald ☝️

Example user-side config:

from prefect.run_configs import ECSRun

# Use the defaults set on the agent
flow.run_config = ECSRun()

# Use the default task definition on the agent, but override cpu and the image
flow.run_config = ECSRun(image="my-fancy-image", cpu="2 vcpu")

# Use an alternate task definition
flow.run_config = ECSRun(task_definition="my-task-definition")

# Override some parameters to run_task
flow.run_config = ECSRun(run_task_kwargs={"overrides": {"executionRoleArn": "..."}})

Great, thanks Jim! You mentioned permissions user, is that coming to Core?

Yeah. By "permissioned" I just mean users that have permission to register task definitions in AWS (which might not be everyone in some teams).

Ah ok, got it. I thought you meant users in the UI 😁 thanks for all the info, looking forward to seeing the changes!

Here's the PR: https://github.com/PrefectHQ/prefect/pull/3585 Note that things don't work quite the way I explained them above (turns out you can't override

image

at runtime, which prefect would require for that model to work). The top comment in the PR should explain how things work as currently written. If any of y'all have feedback on this design, I'd be glad to hear it. The goal here is to make configuring and running prefect on ECS much easier.. cc @Darragh, @Braun Reyes, @Lukas

Thanks for getting back to us with that @Jim Crist-Harif, much appreciated! Having a read now

Yeah, that'd work. Or if you want most flows to use that pattern, you could configure that on the agent as agent-side defaults.

Tried that before on the agent with an older version [0.12.6 I think??] and I could never get it to work. The flow specific one would work better, as we’d be able to configure resources specific to the side car container we’re running 👍

Hopefully the new design makes it easier to configure agent-side defaults, if you ever feel the need to try that pattern. Flow-side configuration is more flexible though, and should handle what you need perfectly.

Thanks Jim, much appreciated! I’ll try carve some time out tomorrow to test it, see if I can make it dance to my tune 😄 Rough idea when the next release is that this PR would be in?

Thanks Darragh. We're off today (election day), but we'll probably issue a release tomorrow.

its been along time since I posted on these threads where I was going to make some changes to make it possible to define a task definition outside of flow registration so that agent does not always attempt to register one.  There was also a change we were going to make to allow for sidecars as well.  Good news is that the new ECSRun and ECSAgent features handles this for you.  Here is a snippet from our deployment process.

ECSRun(
        task_definition_arn=task_definition_arn, run_task_kwargs=dex_run_task_kwargs
    )

Braun Reyes  [11:19 AM] the task definition can have the sidecar defined in it