<@ULVA73B9P> In the 6 last runs of my workflow I'm...
# ask-marvin
g
@Marvin In the 6 last runs of my workflow I'm dealing with an error that crash my runs: Flow run could not be submitted to infrastructure: TaskFailedToStart - Rate limit exceeded while attempting to Create Network Interface in subnet 'subnet-0bc69b119b8266f4d' How can I solve it?
m
thought for 389.0 seconds
g
@Marvin I do not understand
m
thought for 22.5 seconds
No worries — I’m happy to help! Could you tell me what part you’re stuck on? - What are you trying to do with Prefect (run a flow locally, deploy it, use a worker, etc.)? - Are you using Prefect 3.x? - Do you have an error message or a small code snippet I can look at? If you want a super quick starting point, here’s the tiniest Prefect 3.x flow you can run locally:
Copy code
from prefect import flow, get_run_logger

@flow
def say_hi(name="world"):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Hello, {name}!")

if __name__ == "__main__":
    say_hi()
And if you want to deploy that flow from a Git repo in Prefect 3.x:
Copy code
from prefect import flow

# This code is run once to create/update the deployment
flow.from_source(
    source="<https://github.com/owner/repo>",
    entrypoint="path/to/file.py:say_hi",
).deploy(
    name="say-hi",
    work_pool_name="my-work-pool",
)
If you share a bit more about your goal or what’s confusing, I can tailor the explanation and give you the exact steps. For reference, the docs are here: Prefect 3 docs.
g
@Marvin In the 6 last runs of my workflow I'm dealing with an error that crash my runs: Flow run could not be submitted to infrastructure: TaskFailedToStart - Rate limit exceeded while attempting to Create Network Interface in subnet 'subnet-0bc69b119b8266f4d' How can I solve it?
m
thought for 316.7 seconds
Thanks for the details — that error is coming from AWS, not Prefect. It means AWS is throttling ENI (Elastic Network Interface) creation in that specific subnet when your runs start (common with ECS/Fargate and sometimes EKS/EC2). In short: too many tasks/pods are trying to start at once in the same subnet. Here are the most effective mitigations: Prefect-side throttling (quickest fix) - Limit how many runs your worker can start at once. This spreads out ENI creation and avoids the AWS rate limit. - In Prefect 3.x you can set concurrency at the work pool or work queue:
Copy code
# Limit all runs in a work pool
prefect work-pool set-concurrency-limit "<pool-name>" 3

# Or limit a specific queue inside a pool
prefect work-queue set-concurrency-limit "<pool-name>" "<queue-name>" 2
- You can also use a Global Concurrency Limit as a backstop across multiple deployments:
Copy code
prefect gcl create eni-throttle --limit 3
Then attach that limit to your deployments in the UI. Useful links: - Work pool CLI - Work queue CLI AWS-side improvements (to remove the bottleneck) - Spread tasks across multiple subnets/AZs - Configure your ECS worker’s base job template (or deployment
job_variables
) with multiple subnet IDs so new tasks don’t all hit the same subnet’s ENI rate. - Example job variables for an ECS pool/deployment:
Copy code
job_variables:
    networkConfiguration:
      awsvpcConfiguration:
        subnets: ["subnet-aaa", "subnet-bbb", "subnet-ccc"]
        securityGroups: ["sg-xxxx"]
        assignPublicIp: ENABLED
- You can edit this in the Work Pool’s Base Job Template in the UI, or set
job_variables
in your
prefect.yaml
. - ECS guide: Prefect AWS ECS guide - Job variables: Customize job variables - Ensure subnets have enough free IPs and capacity - Use larger CIDR subnets or add additional subnets across AZs and include them in the configuration above. - If you continue to hit ENI API throttling, open an AWS Support case to discuss ENI/API throttling behaviors for your account and region. Operational tips - Stagger schedules or reduce simultaneous scheduled starts, especially on the hour. - If you’re on Fargate and starting many short-lived tasks simultaneously, consider increasing concurrency limits gradually to find a stable threshold for your subnets. A couple of clarifying questions to get you the most precise fix: - Which infrastructure/worker are you using (ECS Fargate, ECS EC2, EKS/Kubernetes, or EC2)? - How many subnets/AZs are configured on your Work Pool or in your deployment
job_variables
? - Do you currently have any work pool or queue concurrency limits set? - Did anything change recently (e.g., more runs scheduled at the same time, new deployments, smaller subnets)? Once I have those details, I can give you an exact configuration example for your setup.
Thanks for the details — this error is coming from AWS when your runs try to start containers and AWS has to create ENIs (network interfaces). When too many tasks start at once in the same subnet, AWS throttles CreateNetworkInterface calls and your flow run fails to start. Here’s how to fix it, starting with the fastest mitigations: 1) Reduce concurrency in Prefect so you don’t hit the AWS ENI rate limit - Set a work pool concurrency limit (affects all workers in that pool):
Copy code
prefect work-pool set-concurrency-limit "<your-ecs-pool-name>" 3
Adjust the number to what your account can handle without throttling, then scale up gradually. Docs: CLI: work-pool - Optionally, add a global concurrency limit for just this flow (by tag) if you only want to throttle specific workloads:
Copy code
prefect gcl create --name eni-protect --limit 3 --tag my-eni-heavy-flows
Docs: Global concurrency limits 2) Spread tasks across multiple subnets/AZs - In your ECS work pool’s base job template, include multiple subnet IDs (ideally across different AZs). This reduces pressure on a single subnet’s ENI creation. - In the Prefect UI: Work Pools -> your ECS pool -> Base job template -> add multiple subnets under
networkConfiguration.awsvpcConfiguration.subnets
. Example snippet:
Copy code
"networkConfiguration": {
  "awsvpcConfiguration": {
    "subnets": ["subnet-aaaa", "subnet-bbbb", "subnet-cccc"],
    "securityGroups": ["sg-xxxxx"],
    "assignPublicIp": "DISABLED"
  }
}
Docs: Prefect ECS guide 3) Make sure you have enough IPs in those subnets - If a subnet is small or nearly full, ENI creation may struggle. Consider using larger CIDR subnets or adding more subnets in the VPC. 4) Consider AWS-side quota/limits - If you routinely launch many tasks at once, request quota increases with AWS Support for the EC2 API request rate/CreateNetworkInterface throttling in your Region. - You can also enable adaptive retry behavior on AWS SDK calls for any code that hits AWS APIs directly: - Set
AWS_RETRY_MODE=adaptive
and
AWS_MAX_ATTEMPTS=10
in the environment for your worker. This won’t fix ECS’s internal calls, but it helps with your own SDK usage. Where to verify - Prefect worker logs for your ECS work pool and the ECS task events in CloudWatch will show the throttling details. Prefect will report “TaskFailedToStart” when ECS can’t start the task due to the ENI rate limit. A couple of quick checks - Are you using an ECS (Fargate) work pool? - How many runs are starting concurrently when this happens? - Which subnets are configured on the pool today (and are they all in one AZ)? If you share your work pool type/name, the current number of subnets, and roughly how many runs kick off at once, I can suggest a safe starting concurrency and the exact base job template edits. Helpful references: - Prefect workers overview - Manage work pools - Work pool CLI