David Martin Calalang
08/14/2025, 4:14 PMprefect.yaml file so that the tasks that are spawned on ECS have those tags?Marvin
08/14/2025, 4:14 PMMarvin
08/14/2025, 4:16 PMjob_variables you set per deployment in prefect.yaml). The ECS worker doesn’t currently expose an AWS tags field, so there’s no prefect.yaml knob that will make spawned ECS tasks carry AWS resource tags.
Example (note the absence of a supported tags key):
deployments:
- name: my-flow
work_pool:
name: my-ecs-pool
job_variables:
cpu: 512
memory: 1024
# no 'tags' support here for ECS tasks
Useful distinctions and options:
- Prefect run tags: You can tag flow runs for filtering/UX in Prefect, but those do not become AWS ECS resource tags.
- AWS-side workarounds:
- If you run via an ECS Service, AWS can propagate tags from the service/task definition to tasks. Prefect’s ECS worker uses RunTask directly (no service), so that path doesn’t help here.
- You could custom-wrap the ECS worker (or fork prefect-aws) to add tags to the RunTask call.
- Post-start tagging via EventBridge/Lambda is possible, if you really enjoy glue.
Docs for context: Prefect + AWS ECS guide
If tagging tasks is critical for cost allocation/compliance in your setup, I can draft a feature request to add tags support to the ECS worker—just say the word and share your tag shape (key/value examples). I’m afraid AWS still won’t read your mind, but we can at least get it to read a dict.David Martin Calalang
08/14/2025, 4:17 PMMarvin
08/14/2025, 4:17 PMMarvin
08/14/2025, 4:18 PMtags field you can set via prefect.yaml or job variables, so the tasks it spawns won’t get your custom AWS tags out of the box.
If tagging the actual ECS Task resources is a must, you’ve got three practical options:
1) Post-launch tagging via EventBridge + Lambda (no Prefect fork)
- Create an EventBridge rule for ECS Task State Change events (e.g., status = RUNNING).
- Have a small Lambda call ecs.tag_resource on the task ARN with your desired tags.
- Pass any context you want to turn into tags (deployment name, environment, cost center) into the container env via Prefect job_variables.env so the Lambda can fetch them via DescribeTasks and construct tags.
Example Lambda (Python):
import os
import boto3
ecs = boto3.client("ecs")
def handler(event, context):
detail = event.get("detail", {})
task_arn = detail.get("taskArn")
cluster_arn = detail.get("clusterArn")
if not task_arn or not cluster_arn:
return
# Pull container overrides/env to derive tags
desc = ecs.describe_tasks(cluster=cluster_arn, tasks=[task_arn])
env = {}
try:
overrides = desc["tasks"][0]["overrides"]["containerOverrides"][0].get("environment", [])
env = {kv["name"]: kv["value"] for kv in overrides}
except Exception:
pass
tags = [
{"key": "Project", "value": env.get("PREFECT_DEPLOYMENT_NAME", "unknown")},
{"key": "Env", "value": env.get("ENVIRONMENT", "unknown")},
{"key": "PrefectRunId", "value": env.get("PREFECT__FLOW_RUN_ID", "unknown")},
]
# Tag the task resource
ecs.tag_resource(resourceArn=task_arn, tags=tags)
EventBridge rule (pattern gist):
- Source: aws.ecs
- Detail type: ECS Task State Change
- Detail.status: RUNNING
Docs:
- ECS TagResource API
- ECS task state change events
2) Enable ECS managed tags (if that’s “good enough”)
- AWS can apply ECS-managed tags to tasks (e.g., cluster, service), but they’re not your custom tags. They require enableECSManagedTags on RunTask. Prefect doesn’t expose that knob today; if/when it does, you’ll get a limited set of AWS-managed tags, not your own.
- RunTask docs for context: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html
3) Add tags support to the ECS worker (needs code change)
- Fork/extend prefect-aws to include a tags field in the call to ecs.run_task and surface it in job variables. Then point your worker to that build.
- Repo: prefect-aws
- If you want, I can open a feature request on your behalf and include your tag use case.
If you share:
- The exact tags you need (keys/values or where values come from)
- Whether a post-launch solution is acceptable
…I’ll tailor a minimal setup (including the job_variables.env to feed the Lambda) or draft the feature request. I can’t make AWS read your mind, but I can make it read a dict.