Short answer: move RegisterTaskDefinition out of the hot path, and throttle the number of concurrent registrations that can happen. Here are the best-practice options in Prefect 3.x for an ECS push work pool: 1) Pre-register a task definition per image build (best mitigation) - During your CI/CD deploy step (where you build/push the image), also call AWS to register a new task definition revision, then pass that ARN to Prefect. When a

task_definition_arn

is set, the ECS worker will not call RegisterTaskDefinition at run time. - Store the ARN in your work pool job variables or directly on the deployment: - Work pool/job variables: set

task_definition_arn

- Deployment via Python:

flow.deploy(
      name="my-deployment",
      work_pool_name="my-ecs-pool",
      variables={"task_definition_arn": "arn:aws:ecs:REGION:ACCT:task-definition/family:rev"}
    )

- Docs: - ECS Worker docs - Customize job variables 2) Limit concurrent flow runs that hit registration at the same time - Work pool concurrency limits cap how many runs your workers pull concurrently from that pool:

prefect work-pool set-concurrency-limit "my-ecs-pool" 5

- If many different deployments are kicking off concurrently (each with a distinct image), this directly reduces simultaneous RegisterTaskDefinition calls. - Docs: - Work pools concept - Manage work pools 3) Reuse existing task definitions when possible - Enable

match_latest_revision_in_family: true

. If the latest active revision in the family matches the desired config, the worker will reuse it instead of registering again. This reduces registrations across worker restarts and between deployments that are effectively identical. - Note: If you change the container image each deploy, equality won’t hold; this helps only when the config (including image) hasn’t changed. - Docs: - ECS Worker docs (job variables) 4) Make caching effective by routing identical runs to the same worker - The ECS worker keeps an in-memory cache of the last compatible task definition per deployment/flow. If multiple workers are polling the same pool/queue, they’ll each end up registering. If feasible, route a given deployment to a dedicated queue (polled by a single worker) so the first run registers and subsequent runs reuse the cached ARN. 5) Soften the impact of throttling with retries/backoff - The ECS worker already retries creating ECS tasks with jitter. You can increase the max attempts and delay via Prefect settings if you’re seeing transient throttling. - If you’d like, I can share the exact environment settings and defaults that control this behavior. Putting it together - If you rebuild the image on each deploy, the cleanest path is to register the task definition at deploy time (CI/CD), then set

task_definition_arn

on the deployment/work pool so runtime never calls RegisterTaskDefinition. - If pre-registration isn’t an option, lower concurrency at the work pool (and optionally isolate deployments with dedicated queues) so you don’t burst RegisterTaskDefinition calls simultaneously. Turn on

match_latest_revision_in_family

for opportunistic reuse. Useful references - ECS push worker and variables: ECS Worker (Prefect AWS) - Serverless compute (ECS) overview: Run flows on serverless compute - Customize job variables: Customize job variables

- Work pools and limits: Work pools and Manage work pools Questions to tailor the recommendation: - Are multiple workers polling the same ECS pool/queue during these bursts? - Can you pre-register TDs in your CI/CD (and inject the ARN into deployment variables)? - Roughly how many distinct deployments/images are starting simultaneously when you see throttling?

Great—yes, you can move RegisterTaskDefinition to your CI/CD and inject the ARN so Prefect won’t register at run-time. Here’s a concrete blueprint you can drop into your pipeline. High-level flow 1) Build and push your image to ECR. 2) Register a new ECS task definition revision that references that image (same family each deploy; new revision). 3) Inject the returned taskDefinitionArn into your Prefect deployment’s job variables (

task_definition_arn

). 4) Deploy/update the Prefect deployment so all runs use the pre-registered TD (no RegisterTaskDefinition during execution). Option A: Programmatic Prefect deploy (recommended for CI) - Step 1–2: Build/push image + register TD (AWS CLI or boto3): Task definition template (commit to repo and substitute IMAGE_URI + params in CI):

{
  "family": "my-family",
  "networkMode": "awsvpc",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::123456789012:role/myTaskRole",
  "containerDefinitions": [
    {
      "name": "flow",
      "image": "IMAGE_URI",
      "essential": true,
      "command": [],
      "environment": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/my-family",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

Register the TD (CLI):

aws ecs register-task-definition --cli-input-json <file://taskdef.json> \
  --query 'taskDefinition.taskDefinitionArn' --output text

Capture the ARN in a CI var like

TD_ARN

. - Step 3–4: Deploy/update your Prefect deployment with the ARN: Small deploy script in your repo:

from prefect import flow
from prefect.docker import DockerImage

from flows.my_flow import my_flow  # your flow import

TD_ARN = "<inject at runtime>"      # e.g., from env
IMAGE = "<your ecr repo>:<tag>"     # e.g., from env

if __name__ == "__main__":
    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-ecs-pool",
        image=DockerImage(name=IMAGE),
        variables={
            "task_definition_arn": TD_ARN
        },
    )

In CI: - Export env vars for

TD_ARN

and

IMAGE

. - Run

python deploy.py

(or

uv run

if you prefer). - This updates/creates the deployment pointing at your pre-registered TD. Option B: YAML + prefect deploy (no Python script) - Keep your

prefect.yaml

under version control and patch

job_variables.task_definition_arn

at build time, then run

prefect deploy

. prefect.yaml snippet:

deployments:
- name: my-deployment
  entrypoint: flows/my_flow.py:my_flow
  work_pool:
    name: my-ecs-pool
    job_variables:
      task_definition_arn: arn:aws:ecs:us-east-1:123456789012:task-definition/my-family:42
  tags: []
  description: ""

In CI: - Build/push image. - Register TD, capture ARN. - Patch the YAML with the new ARN (e.g., using yq/sed). - Run:

prefect deploy

This will create/update the deployment with the new ARN. Notes and guardrails - One family per deployment is a clean pattern: each deploy registers a new revision in that family, and you just update the deployment’s

task_definition_arn

to the latest. - If you really want to set a default at the work pool level, you can do that in the UI (Work pool > Edit > Default job variables). But that forces every deployment in the pool to share the same TD—usually not what you want if each deployment has its own image. - Since you’re hitting throttling around ~100 concurrent runs, pre-registering will remove RegisterTaskDefinition from the hot path entirely. As an extra guard, you can still cap burstiness with a work-pool concurrency limit:

prefect work-pool set-concurrency-limit "my-ecs-pool" 50

You can tune this up once pre-registration is in place. Docs and references

- ECS worker variables (including

task_definition_arn

): ECS Worker (Prefect AWS) - Serverless/ECS overview + patterns: Run flows on serverless compute - Overriding job variables on deployments: Customize job variables - Work pool limits (CLI verified): Manage work pools If you share what CI system you’re using (GitHub Actions, GitLab CI, Jenkins, etc.), I can give you a drop-in pipeline example with the exact steps and environment wiring.