<@ULVA73B9P> I'm deploying flows to a work-pool th...
# ask-marvin
d
@Marvin I'm deploying flows to a work-pool that runs tasks on ECS. Is there a way to add tags to my
prefect.yaml
file so that the tasks that are spawned on ECS have those tags?
m
thought for 103.4 seconds
Short answer: not today. In Prefect 3.x, ECS task settings come from the ECS work pool’s job variables (and any
job_variables
you set per deployment in
prefect.yaml
). The ECS worker doesn’t currently expose an AWS
tags
field, so there’s no
prefect.yaml
knob that will make spawned ECS tasks carry AWS resource tags. Example (note the absence of a supported
tags
key):
Copy code
deployments:
  - name: my-flow
    work_pool:
      name: my-ecs-pool
    job_variables:
      cpu: 512
      memory: 1024
      # no 'tags' support here for ECS tasks
Useful distinctions and options: - Prefect run tags: You can tag flow runs for filtering/UX in Prefect, but those do not become AWS ECS resource tags. - AWS-side workarounds: - If you run via an ECS Service, AWS can propagate tags from the service/task definition to tasks. Prefect’s ECS worker uses RunTask directly (no service), so that path doesn’t help here. - You could custom-wrap the ECS worker (or fork
prefect-aws
) to add
tags
to the
RunTask
call. - Post-start tagging via EventBridge/Lambda is possible, if you really enjoy glue. Docs for context: Prefect + AWS ECS guide If tagging tasks is critical for cost allocation/compliance in your setup, I can draft a feature request to add
tags
support to the ECS worker—just say the word and share your tag shape (key/value examples). I’m afraid AWS still won’t read your mind, but we can at least get it to read a dict.
d
@Marvin So if I wanted to tag the spawned ECS tasks, how would I do so?
m
thought for 100.7 seconds
Short version: Prefect’s ECS worker doesn’t currently expose a
tags
field you can set via
prefect.yaml
or job variables, so the tasks it spawns won’t get your custom AWS tags out of the box. If tagging the actual ECS Task resources is a must, you’ve got three practical options: 1) Post-launch tagging via EventBridge + Lambda (no Prefect fork) - Create an EventBridge rule for ECS Task State Change events (e.g., status = RUNNING). - Have a small Lambda call
ecs.tag_resource
on the task ARN with your desired tags. - Pass any context you want to turn into tags (deployment name, environment, cost center) into the container env via Prefect
job_variables.env
so the Lambda can fetch them via
DescribeTasks
and construct tags. Example Lambda (Python):
Copy code
import os
import boto3

ecs = boto3.client("ecs")

def handler(event, context):
    detail = event.get("detail", {})
    task_arn = detail.get("taskArn")
    cluster_arn = detail.get("clusterArn")
    if not task_arn or not cluster_arn:
        return

    # Pull container overrides/env to derive tags
    desc = ecs.describe_tasks(cluster=cluster_arn, tasks=[task_arn])
    env = {}
    try:
        overrides = desc["tasks"][0]["overrides"]["containerOverrides"][0].get("environment", [])
        env = {kv["name"]: kv["value"] for kv in overrides}
    except Exception:
        pass

    tags = [
        {"key": "Project", "value": env.get("PREFECT_DEPLOYMENT_NAME", "unknown")},
        {"key": "Env", "value": env.get("ENVIRONMENT", "unknown")},
        {"key": "PrefectRunId", "value": env.get("PREFECT__FLOW_RUN_ID", "unknown")},
    ]

    # Tag the task resource
    ecs.tag_resource(resourceArn=task_arn, tags=tags)
EventBridge rule (pattern gist): - Source: aws.ecs - Detail type: ECS Task State Change - Detail.status: RUNNING Docs: - ECS TagResource API - ECS task state change events 2) Enable ECS managed tags (if that’s “good enough”) - AWS can apply ECS-managed tags to tasks (e.g., cluster, service), but they’re not your custom tags. They require
enableECSManagedTags
on RunTask. Prefect doesn’t expose that knob today; if/when it does, you’ll get a limited set of AWS-managed tags, not your own. - RunTask docs for context: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html 3) Add
tags
support to the ECS worker (needs code change) - Fork/extend
prefect-aws
to include a
tags
field in the call to
ecs.run_task
and surface it in job variables. Then point your worker to that build. - Repo: prefect-aws - If you want, I can open a feature request on your behalf and include your tag use case. If you share: - The exact tags you need (keys/values or where values come from) - Whether a post-launch solution is acceptable …I’ll tailor a minimal setup (including the
job_variables.env
to feed the Lambda) or draft the feature request. I can’t make AWS read your mind, but I can make it read a dict.