David Martin Calalang
08/14/2025, 4:14 PMprefect.yaml
file so that the tasks that are spawned on ECS have those tags?Marvin
08/14/2025, 4:14 PMMarvin
08/14/2025, 4:16 PMjob_variables
you set per deployment in prefect.yaml
). The ECS worker doesn’t currently expose an AWS tags
field, so there’s no prefect.yaml
knob that will make spawned ECS tasks carry AWS resource tags.
Example (note the absence of a supported tags
key):
deployments:
- name: my-flow
work_pool:
name: my-ecs-pool
job_variables:
cpu: 512
memory: 1024
# no 'tags' support here for ECS tasks
Useful distinctions and options:
- Prefect run tags: You can tag flow runs for filtering/UX in Prefect, but those do not become AWS ECS resource tags.
- AWS-side workarounds:
- If you run via an ECS Service, AWS can propagate tags from the service/task definition to tasks. Prefect’s ECS worker uses RunTask directly (no service), so that path doesn’t help here.
- You could custom-wrap the ECS worker (or fork prefect-aws
) to add tags
to the RunTask
call.
- Post-start tagging via EventBridge/Lambda is possible, if you really enjoy glue.
Docs for context: Prefect + AWS ECS guide
If tagging tasks is critical for cost allocation/compliance in your setup, I can draft a feature request to add tags
support to the ECS worker—just say the word and share your tag shape (key/value examples). I’m afraid AWS still won’t read your mind, but we can at least get it to read a dict.David Martin Calalang
08/14/2025, 4:17 PMMarvin
08/14/2025, 4:17 PMMarvin
08/14/2025, 4:18 PMtags
field you can set via prefect.yaml
or job variables, so the tasks it spawns won’t get your custom AWS tags out of the box.
If tagging the actual ECS Task resources is a must, you’ve got three practical options:
1) Post-launch tagging via EventBridge + Lambda (no Prefect fork)
- Create an EventBridge rule for ECS Task State Change events (e.g., status = RUNNING).
- Have a small Lambda call ecs.tag_resource
on the task ARN with your desired tags.
- Pass any context you want to turn into tags (deployment name, environment, cost center) into the container env via Prefect job_variables.env
so the Lambda can fetch them via DescribeTasks
and construct tags.
Example Lambda (Python):
import os
import boto3
ecs = boto3.client("ecs")
def handler(event, context):
detail = event.get("detail", {})
task_arn = detail.get("taskArn")
cluster_arn = detail.get("clusterArn")
if not task_arn or not cluster_arn:
return
# Pull container overrides/env to derive tags
desc = ecs.describe_tasks(cluster=cluster_arn, tasks=[task_arn])
env = {}
try:
overrides = desc["tasks"][0]["overrides"]["containerOverrides"][0].get("environment", [])
env = {kv["name"]: kv["value"] for kv in overrides}
except Exception:
pass
tags = [
{"key": "Project", "value": env.get("PREFECT_DEPLOYMENT_NAME", "unknown")},
{"key": "Env", "value": env.get("ENVIRONMENT", "unknown")},
{"key": "PrefectRunId", "value": env.get("PREFECT__FLOW_RUN_ID", "unknown")},
]
# Tag the task resource
ecs.tag_resource(resourceArn=task_arn, tags=tags)
EventBridge rule (pattern gist):
- Source: aws.ecs
- Detail type: ECS Task State Change
- Detail.status: RUNNING
Docs:
- ECS TagResource API
- ECS task state change events
2) Enable ECS managed tags (if that’s “good enough”)
- AWS can apply ECS-managed tags to tasks (e.g., cluster, service), but they’re not your custom tags. They require enableECSManagedTags
on RunTask. Prefect doesn’t expose that knob today; if/when it does, you’ll get a limited set of AWS-managed tags, not your own.
- RunTask docs for context: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html
3) Add tags
support to the ECS worker (needs code change)
- Fork/extend prefect-aws
to include a tags
field in the call to ecs.run_task
and surface it in job variables. Then point your worker to that build.
- Repo: prefect-aws
- If you want, I can open a feature request on your behalf and include your tag use case.
If you share:
- The exact tags you need (keys/values or where values come from)
- Whether a post-launch solution is acceptable
…I’ll tailor a minimal setup (including the job_variables.env
to feed the Lambda) or draft the feature request. I can’t make AWS read your mind, but I can make it read a dict.