Braun Reyes
10/26/2020, 7:20 PMextra_containers
argument to support adding sidecars to the flow run container
2 - add the ability to skip the task definition registration based a run context key of task_arn_override
. This would allow folks using a map to submit flow run(subflow pattern) to skip the task registration issue that @Darragh and @Lukas were running intoDarragh
10/26/2020, 7:25 PMBraun Reyes
10/26/2020, 7:27 PMDarragh
10/26/2020, 7:33 PMjosh
10/26/2020, 7:44 PMBraun Reyes
10/26/2020, 7:45 PMJim Crist-Harif
10/26/2020, 7:56 PMfamily
, family:revision
, or full task def arn). If not provided, the default task definition name set on the agent will be used.
• Any "overrides" for that task definition to set at task startup. This includes support for basically anything you can pass to run_task
(https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task). Some common parameters like image
/`cpu` /`memory` /etc... have been pulled out for easy access, but users can configure anything to pass to run_task
on the flow itself.
The ECS agent is then configured with a default task definition to use (if not provided, it will auto-register a basic definition that ships with prefect). No other task definitions will be auto-registered by prefect, if you want to provide new custom definitions you'd need to register those yourself. To simplify registering custom task definitions for users, we'll also provide a helper in the prefect cli as prefect agent ecs register-task-definitions ...
.
This means that no definitions are registered at agent runtime, all the agent does is all run_task
with an already existing definition and a set of configurable parameters (either configurable on the flow or the agent, with the flow taking precedence).
The hope is that this design can support everything a user/team might want, while also being much simpler to maintain and reason about.Braun Reyes
10/26/2020, 7:58 PMJim Crist-Harif
10/26/2020, 7:59 PMECSAgent
will only support flows configured with a flow.run_config
(not a flow.environment
). Depending on how fast you think you can transition over to the new model, it might still be worth your time to patch the old fargate agent.Braun Reyes
10/26/2020, 8:00 PMLukas
10/26/2020, 8:51 PMjosh
10/26/2020, 9:01 PMDarragh
10/26/2020, 9:06 PMJim Crist-Harif
10/26/2020, 9:09 PMDarragh
10/26/2020, 9:12 PMJim Crist-Harif
10/26/2020, 9:15 PMtask_definition
to ECSRun
, or if all flows should use it an admin would configure the ecs agent to use it as the default definition.Braun Reyes
10/26/2020, 9:19 PMJim Crist-Harif
10/26/2020, 9:19 PMfrom prefect.run_configs import ECSRun
# Use the defaults set on the agent
flow.run_config = ECSRun()
# Use the default task definition on the agent, but override cpu and the image
flow.run_config = ECSRun(image="my-fancy-image", cpu="2 vcpu")
# Use an alternate task definition
flow.run_config = ECSRun(task_definition="my-task-definition")
# Override some parameters to run_task
flow.run_config = ECSRun(run_task_kwargs={"overrides": {"executionRoleArn": "..."}})
Darragh
10/26/2020, 9:22 PMJim Crist-Harif
10/26/2020, 9:23 PMDarragh
10/26/2020, 9:47 PMJim Crist-Harif
10/28/2020, 10:57 PMimage
at runtime, which prefect would require for that model to work). The top comment in the PR should explain how things work as currently written.
If any of y'all have feedback on this design, I'd be glad to hear it. The goal here is to make configuring and running prefect on ECS much easier..
cc @Darragh, @Braun Reyes, @LukasDarragh
10/29/2020, 5:10 PMECSRun(task_definition={containers: [{image: my_first_container}, {image: my_second_container, mount_volume: my_first_container/SOME_PATH]})
Jim Crist-Harif
10/30/2020, 4:28 PMDarragh
10/30/2020, 4:42 PMJim Crist-Harif
10/30/2020, 6:14 PMDarragh
10/31/2020, 7:52 PMJim Crist-Harif
11/03/2020, 3:06 PMBraun Reyes
01/22/2021, 5:21 PMECSRun(
task_definition_arn=task_definition_arn, run_task_kwargs=dex_run_task_kwargs
)
Braun Reyes [11:19 AM]
the task definition can have the sidecar defined in it