Thanks for the details — these two errors point to two separate root causes in an ECS push setup: 1) ResourceInitializationError pulling a secret from Secrets Manager - Root cause: your ECS task can’t reach AWS Secrets Manager (networking), not an IAM deny. The “StatusCode: 0 … context deadline exceeded” almost always means the task has no egress path to AWS services. - Fix options: - If you’re using private subnets, add either: - NAT gateway + proper route tables, or - VPC interface endpoints for: - com.amazonaws.eu-central-1.secretsmanager - com.amazonaws.eu-central-1.ecr.api - com.amazonaws.eu-central-1.ecr.dkr - com.amazonaws.eu-central-1.logs - (recommended) com.amazonaws.eu-central-1.sts - And a gateway endpoint for S3 (ECR layers are fetched from S3) - If you’re using public subnets, set the work pool to assign a public IP and ensure the subnet has an Internet Gateway: - Job variables: set

network_configuration

with your subnets/security groups and

assignPublicIp: ENABLED

- Ensure the security group attached to the task ENI allows outbound TCP 443 (and NACLs are not blocking). - Confirm the execution role has permission to read the secret and KMS decrypt if the secret uses a CMK (see IAM section below). Helpful AWS docs: - ECS task networking - Secrets Manager VPC endpoints - ECR VPC endpoints - CloudWatch Logs interface endpoints 2) AccessDenied on ecs:DescribeTaskDefinition for the push caller - Root cause: the AWS identity used by your Prefect ECS push work pool (looks like IAM user

prefect-ecs-push-user

) is missing at least

ecs:DescribeTaskDefinition

. In practice you’ll also need other ECS calls and

iam:PassRole

to pass the execution/task roles. What to change (IAM) A) Caller identity (the AWS credentials configured on your ECS push work pool) Grant the Prefect “submitter” identity these minimum permissions. Replace ACCOUNT_ID with 764446573600 and REGION with eu-central-1 if you want to scope tighter.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PrefectECSPushCore",
      "Effect": "Allow",
      "Action": [
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:DescribeTasks",
        "ecs:DescribeClusters",
        "ecs:ListClusters",
        "ecs:RegisterTaskDefinition",
        "ecs:DeregisterTaskDefinition",
        "ecs:DescribeTaskDefinition",
        "ecs:ListTaskDefinitions",
        "ecs:TagResource",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "logs:DescribeLogGroups"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowPassExecutionAndTaskRoles",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::764446573600:role/PrefectEcsTaskExecutionRole",
        "arn:aws:iam::764446573600:role/PrefectECSTaskRole"
      ]
    }
  ]
}

B) ECS Task Execution role (used by the running task to pull image, fetch secrets, write logs) Attach the AWS managed policy plus explicit Secrets Manager (and KMS if needed): - Attach: arnawsiam:awspolicy/service-role/AmazonECSTaskExecutionRolePolicy - Add this inline: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "SecretsManagerAccess", "Effect": "Allow", "Action": ["secretsmanager:GetSecretValue"], "Resource": "arnawssecretsmanagereu central 1764446573600㊙️prefect/production/*" }, { "Sid": "KmsDecryptForSecrets", "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arnawskmseu central 1764446573600:key/*",

"Condition": { "StringEquals": { "kms:ViaService": "secretsmanager.eu-central-1.amazonaws.com" } } } ] }

Note: The managed execution role policy already covers ECR and CloudWatch Logs. If you disabled CloudWatch logs in your pool, the logs permissions are optional.

C) Optional ECS Task role
Only needed if your flows call AWS (e.g., S3). Attach the permissions your code needs and reference this role in the work pool/deployment as `task_role_arn`.

Prefect ECS push configuration checks

- Inspect your pool to confirm what Prefect is sending to ECS (cluster, subnets, security groups, execution role, etc.):

prefect work-pool inspect "<your-ecs-push-pool>" ``` - Key job variables you’ll typically set on an ECS push pool: -

cluster

(name or ARN) -

execution_role_arn

-

task_role_arn

(optional, for flow’s AWS access) -

vpc_id

and/or

network_configuration

(subnets, security_groups, assignPublicIp) -

configure_cloudwatch_logs

(true/false),

cloudwatch_logs_prefix

Docs: - Serverless (ECS push) guide - Customize job variables - Work pools concepts About the 300s timeout - The “Reached configured timeout of 300s” is a symptom of the task never becoming healthy because it can’t fetch the secret. Fixing the networking/IAM for Secrets Manager will resolve the timeout. In Prefect 3.x, the legacy per-task timeout job variables are deprecated; focus on correcting connectivity and permissions. Quick checklist for your specific errors - Execution role can read the secret: - Has

secretsmanager:GetSecretValue

for arnawssecretsmanagereu central 1764446573600㊙️prefect/production/api-key-... - If secret is CMK-encrypted, has

kms:Decrypt

for that key - Task network can reach AWS services: - Public subnets + assignPublicIp: ENABLED, or - Private subnets + NAT, or - Private subnets + VPC endpoints (Secrets Manager, ECR api/dkr, Logs, S3 gateway, STS) - Security group egress allows TCP 443 - Caller identity (prefect-ecs-push-user) has: -

ecs:DescribeTaskDefinition

(fix your current AccessDenied) -

ecs:RunTask

,

ecs:Describe*

-

iam:PassRole

for your execution/task roles If you can share: - The output of

prefect work-pool inspect "<pool>"

(redact sensitive fields) - Whether your subnets are public or private and if

assignPublicIp

is enabled - The execution role ARN you’ve configured …I can pinpoint exactly which setting is blocking you and provide the exact JSON you need.

Short answer: yes—you can use AWS Systems Manager Parameter Store instead of Secrets Manager. Many teams do this to avoid creating a VPC interface endpoint for Secrets Manager in private subnets. Key points and trade-offs: - Network: If your workloads run in private subnets without NAT, you’ll need VPC interface endpoints for whichever service you use. - Using Secrets Manager → add

secretsmanager

(and typically

kms

for decryption). - Using Parameter Store → add

ssm

(and

kms

for SecureString decryption). If you assume roles, you may also need a regional

sts

endpoint. - Features: Secrets Manager has built-in rotation and higher size limits (up to 64 KB). Parameter Store is cheaper but has 4–8 KB limits and no built-in rotation (you can roll your own with Lambda/EventBridge). - Prefect integration: In Prefect 3.x,

prefect-aws

includes helpers for Secrets Manager, but not for Parameter Store. If you choose Parameter Store, just use boto3 in your flow/task. Example: fetching a SecureString from SSM Parameter Store in a Prefect flow

from prefect import flow, task
import boto3

@task
def get_param(name: str, region: str | None = None) -> str:
    ssm = boto3.client("ssm", region_name=region)
    resp = ssm.get_parameter(Name=name, WithDecryption=True)
    return resp["Parameter"]["Value"]

@flow
def main():
    password = get_param("/myapp/prod/db/password")
    # use the password safely
    return "ok"

if __name__ == "__main__":
    main()

Required IAM permissions for the runtime role: -

ssm:GetParameter

(and/or

ssm:GetParametersByPath

if you need it) -

kms:Decrypt

for the KMS key used to encrypt SecureString Questions to make sure this fits your setup: - Are your Prefect runs executing on ECS, Kubernetes, or EC2 in private subnets without NAT? - Do you already have VPC endpoints for

ssm

and

kms

, or are you planning to add them?

Got it — you’re on ECS and don’t have any VPC endpoints yet. Key takeaway: switching to SSM Parameter Store does not eliminate the need for network egress. Your ECS tasks still need a path to AWS APIs. So you have two viable patterns: Option 1: Use a NAT gateway (simplest to start) - Works with either SSM Parameter Store or Secrets Manager without creating any VPC endpoints. - Pros: fastest path to “it works.” - Cons: ongoing NAT cost and broader Internet egress. - What to do: - Put ECS tasks in private subnets with a route to a NAT Gateway. - Give the task/execution role these permissions if using SSM SecureString: - ssm:GetParameter (and/or ssm:GetParametersByPath) - kms:Decrypt - Optional: inject secrets directly via ECS task definition “secrets” from SSM. Option 2: Stay fully private (no Internet) with VPC interface endpoints - You’ll need endpoints for whichever services your tasks call. For ECS + SSM, the typical minimum set is: - com.amazonaws.<region>.ssm (Parameter Store) - com.amazonaws.<region>.kms (for SecureString decryption) - com.amazonaws.<region>.sts (if you assume roles or need temporary creds) - If you pull images from ECR: com.amazonaws.<region>.ecr.api and com.amazonaws.<region>.ecr.dkr - If you send logs to CloudWatch: com.amazonaws.<region>.logs - Add others as your stack requires (e.g., S3 gateway endpoint if you read code or data from S3). - You’ll also need the right security group rules and endpoint policies to allow these calls. ECS-native secret injection with SSM - You can avoid writing boto3 code by letting ECS inject SSM parameters into container env vars: - In your task definition, add a “secrets” entry like:

"secrets": [
      { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:REGION:ACCOUNT_ID:parameter/myapp/prod/db/password" }
    ]

- Ensure the task execution role has

ssm:GetParameter

and

kms:Decrypt

(if SecureString). - Network connectivity requirement still applies (via NAT or the endpoints above). Important Prefect-specific note - If you’re using Prefect Cloud, your worker/tasks must reach Prefect’s public API. With “no Internet” you’ll need NAT (or a corporate proxy) regardless of SSM vs Secrets Manager. If you’re running self-hosted Prefect Server inside your VPC, you can stay fully private. A couple quick questions to tailor the exact checklist: - Are your ECS tasks/Fargate running in private subnets without NAT today? - Are you using Prefect Cloud or a self-hosted Prefect Server? - Are your container images in ECR and logs in CloudWatch Logs? (If yes and you want no Internet, you’ll need the ECR and Logs endpoints too.)