Gastón Peña
10/20/2025, 8:24 PMMarvin
10/20/2025, 8:25 PMGastón Peña
10/20/2025, 8:28 PMMarvin
10/20/2025, 8:28 PMMarvin
10/20/2025, 8:29 PMfrom prefect import flow, get_run_logger
@flow
def say_hi(name="world"):
logger = get_run_logger()
<http://logger.info|logger.info>(f"Hello, {name}!")
if __name__ == "__main__":
say_hi()
And if you want to deploy that flow from a Git repo in Prefect 3.x:
from prefect import flow
# This code is run once to create/update the deployment
flow.from_source(
source="<https://github.com/owner/repo>",
entrypoint="path/to/file.py:say_hi",
).deploy(
name="say-hi",
work_pool_name="my-work-pool",
)
If you share a bit more about your goal or what’s confusing, I can tailor the explanation and give you the exact steps. For reference, the docs are here: Prefect 3 docs.Gastón Peña
10/20/2025, 8:29 PMMarvin
10/20/2025, 8:29 PMMarvin
10/20/2025, 8:31 PM# Limit all runs in a work pool
prefect work-pool set-concurrency-limit "<pool-name>" 3
# Or limit a specific queue inside a pool
prefect work-queue set-concurrency-limit "<pool-name>" "<queue-name>" 2
- You can also use a Global Concurrency Limit as a backstop across multiple deployments:
prefect gcl create eni-throttle --limit 3
Then attach that limit to your deployments in the UI.
Useful links:
- Work pool CLI
- Work queue CLI
AWS-side improvements (to remove the bottleneck)
- Spread tasks across multiple subnets/AZs
- Configure your ECS worker’s base job template (or deployment job_variables) with multiple subnet IDs so new tasks don’t all hit the same subnet’s ENI rate.
- Example job variables for an ECS pool/deployment:
job_variables:
networkConfiguration:
awsvpcConfiguration:
subnets: ["subnet-aaa", "subnet-bbb", "subnet-ccc"]
securityGroups: ["sg-xxxx"]
assignPublicIp: ENABLED
- You can edit this in the Work Pool’s Base Job Template in the UI, or set job_variables in your prefect.yaml.
- ECS guide: Prefect AWS ECS guide
- Job variables: Customize job variables
- Ensure subnets have enough free IPs and capacity
- Use larger CIDR subnets or add additional subnets across AZs and include them in the configuration above.
- If you continue to hit ENI API throttling, open an AWS Support case to discuss ENI/API throttling behaviors for your account and region.
Operational tips
- Stagger schedules or reduce simultaneous scheduled starts, especially on the hour.
- If you’re on Fargate and starting many short-lived tasks simultaneously, consider increasing concurrency limits gradually to find a stable threshold for your subnets.
A couple of clarifying questions to get you the most precise fix:
- Which infrastructure/worker are you using (ECS Fargate, ECS EC2, EKS/Kubernetes, or EC2)?
- How many subnets/AZs are configured on your Work Pool or in your deployment job_variables?
- Do you currently have any work pool or queue concurrency limits set?
- Did anything change recently (e.g., more runs scheduled at the same time, new deployments, smaller subnets)?
Once I have those details, I can give you an exact configuration example for your setup.Marvin
10/20/2025, 8:34 PMprefect work-pool set-concurrency-limit "<your-ecs-pool-name>" 3
Adjust the number to what your account can handle without throttling, then scale up gradually. Docs: CLI: work-pool
- Optionally, add a global concurrency limit for just this flow (by tag) if you only want to throttle specific workloads:
prefect gcl create --name eni-protect --limit 3 --tag my-eni-heavy-flows
Docs: Global concurrency limits
2) Spread tasks across multiple subnets/AZs
- In your ECS work pool’s base job template, include multiple subnet IDs (ideally across different AZs). This reduces pressure on a single subnet’s ENI creation.
- In the Prefect UI: Work Pools -> your ECS pool -> Base job template -> add multiple subnets under networkConfiguration.awsvpcConfiguration.subnets.
Example snippet:
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": ["subnet-aaaa", "subnet-bbbb", "subnet-cccc"],
"securityGroups": ["sg-xxxxx"],
"assignPublicIp": "DISABLED"
}
}
Docs: Prefect ECS guide
3) Make sure you have enough IPs in those subnets
- If a subnet is small or nearly full, ENI creation may struggle. Consider using larger CIDR subnets or adding more subnets in the VPC.
4) Consider AWS-side quota/limits
- If you routinely launch many tasks at once, request quota increases with AWS Support for the EC2 API request rate/CreateNetworkInterface throttling in your Region.
- You can also enable adaptive retry behavior on AWS SDK calls for any code that hits AWS APIs directly:
- Set AWS_RETRY_MODE=adaptive and AWS_MAX_ATTEMPTS=10 in the environment for your worker. This won’t fix ECS’s internal calls, but it helps with your own SDK usage.
Where to verify
- Prefect worker logs for your ECS work pool and the ECS task events in CloudWatch will show the throttling details. Prefect will report “TaskFailedToStart” when ECS can’t start the task due to the ENI rate limit.
A couple of quick checks
- Are you using an ECS (Fargate) work pool?
- How many runs are starting concurrently when this happens?
- Which subnets are configured on the pool today (and are they all in one AZ)?
If you share your work pool type/name, the current number of subnets, and roughly how many runs kick off at once, I can suggest a safe starting concurrency and the exact base job template edits.
Helpful references:
- Prefect workers overview
- Manage work pools
- Work pool CLI