@Marvin I'm running a flow on an ecs hosted worker. One thing that it does is kicks off some sub flows specifically using run_deployment (we want them to be seaparate). I get an error from the prefect_cloud.push_work_pool logger:

Could not load job configuration: Client error '429 Too Many Requests' for url '<http://orion-internal:4200/api/accounts/xxxxxx/workspaces/yyyyyy/block_documents/zzzzzzz?include_secrets=true>'
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429>

This block id is our aws credentials used for starting new jobs. Why is the host name for this "orion-internal"? Is this a log message coming from the prefect cloud infrastructure or our hosted runner? if it was on our hosted runner then I expect that it would have shown api.prefect.cloud instead. What is the rate limit for this API?

Great questions — here’s what’s going on and how to address it. What’s generating the log and why “orion-internal” - The logger name

prefect_cloud.push_work_pool

indicates this message is coming from Prefect Cloud’s push work pool service, not from your ECS-hosted worker. - The hostname

orion-internal:4200

is an internal DNS name used inside Prefect Cloud’s infrastructure to talk to the Prefect API service. It’s expected to see that in Cloud-originated logs and does not indicate a misconfiguration on your side. - This typically happens when the sub-deployments you trigger with

run_deployment

are targeting a push work pool (or any pool handled by Prefect Cloud directly), even if the parent flow is running on your ECS worker. Each deployment is independent and uses its own

work_pool_name

. Why you’re seeing 429 Too Many Requests - When a deployment is picked up (by a push pool service or a worker), Prefect resolves any block references in the job configuration — e.g., your AWS Credentials block — via API calls like

/block_documents/{id}?include_secrets=true

. - If you kick off many sub-deployments concurrently with

run_deployment

, each one will fetch those blocks, which can generate a burst of API requests and hit your account’s API request rate limit. Prefect will retry 429s with backoff, but sufficiently high burst/concurrency can still surface errors. Prefect Cloud rate limits - Rate limits apply at the account level and block document reads count toward the general API Requests bucket. - Per-minute limits (API Requests | Logs+Events): - Hobby: 625 | 2,000 - Starter: 1,250 | 2,800 - Team: 2,500 | 8,000 - Pro: 5,000 | 40,000 - Enterprise: custom - Docs: Rate limits and data retention How to mitigate - Ensure sub-deployments target your ECS work pool (pull/worker-based) if you want them handled by your worker instead of push work pools. That will also make logs show

api.prefect.cloud

from your side. Double-check each sub-deployment’s

work_pool_name

. - Reduce fan-out/concurrency of

run_deployment

calls: - Add a tag to those deployments and set a tag-based concurrency limit in Prefect so only N sub-deployments start at a time. - Stagger/batch sub-deployments if possible. - Prefer IAM roles over AWS Credentials blocks for ECS tasks. If your task definitions/work pool can assume an IAM role, you can avoid resolving an AWS Credentials block (and its include_secrets fetch) on every run. - If you’re consistently near limits, consider upgrading your plan or contact us about increasing limits. A couple of quick checks/questions - Which

work_pool_name

do the sub-deployments use? If it’s a push pool, that explains the

prefect_cloud.push_work_pool

log and

orion-internal

hostname. - Roughly how many

run_deployment

calls are firing concurrently? - Which Prefect Cloud plan is your account on? We can sanity-check your expected request volume against your plan’s limits. - If helpful, I can suggest a concrete tagging + concurrency-limit setup for your sub-deployments.