Jeandre de Bruyn
12/03/2025, 1:20 PMMarvin
12/03/2025, 1:20 PMMarvin
12/03/2025, 1:32 PMaws_credentials job variable. It’s receiving an empty object {}, so Pydantic is treating aws_access_key_id, aws_secret_access_key, and region_name as missing.
What this usually means
- The work pool’s Base job template has aws_credentials: {} (empty), or
- Your deployment overrides job_variables.aws_credentials: {} (empty), or
- A block reference was mis-specified so it resolved to {} instead of a valid block.
How to fix it (pick one)
A) Easiest: set AWS credentials on the work pool (UI)
- In Prefect: Work Pools → your ECS push pool → Settings
- Set “AWS Credentials” to an AwsCredentials block (create one if needed)
- Set region (recommended for push pools)
- Save
Docs:
- Work pools (push/serverless) overview: Serverless work pools
- Managing Base job templates: Manage work pools
B) CLI: attach a credentials block in the Base job template
- Get the default template and edit it to point at your block:
prefect work-pool get-default-base-job-template --type ecs:push > ecs_base.json
Edit ecs_base.json so the aws_credentials variable default is a block reference:
{
"variables": {
"type": "object",
"properties": {
"aws_credentials": {
"title": "AWS Credentials",
"type": "object",
"default": {
"$ref": {
"block_document_slug": "aws-credentials/my-aws-creds"
}
}
},
"region_name": {
"type": "string",
"default": "us-east-1"
}
}
}
}
Then update the pool:
prefect work-pool update my-ecs-pool --base-job-template ecs_base.json
C) Per-deployment override (Python or prefect.yaml)
- Python:
from prefect import flow
from prefect_aws import AwsCredentials
@flow
def my_flow():
...
creds = AwsCredentials.load("my-aws-creds")
if __name__ == "__main__":
my_flow.deploy(
name="ecs-deploy",
work_pool_name="my-ecs-pool",
job_variables={
"aws_credentials": {"$ref": {"block_document_slug": "aws-credentials/my-aws-creds"}},
"region_name": "us-east-1"
},
)
- prefect.yaml:
deployments:
- name: ecs-deploy
entrypoint: flow.py:my_flow
work_pool:
name: my-ecs-pool
job_variables:
aws_credentials:
$ref:
block_document_slug: "aws-credentials/my-aws-creds"
region_name: "us-east-1"
Creating the AwsCredentials block
- You can use static keys, a profile, or ambient credentials. Region is strongly recommended for push pools.
from prefect_aws import AwsCredentials
# Static keys
AwsCredentials(
aws_access_key_id="AKIA...",
aws_secret_access_key="...",
region_name="us-east-1",
).save("my-aws-creds", overwrite=True)
# Or use an AWS CLI profile
AwsCredentials(
profile_name="my-profile",
region_name="us-east-1",
).save("my-aws-creds", overwrite=True)
Docs:
- AWS integration and credentials block: prefect-aws integration
- Customizing job variables (including block refs): Customize job variables
Quick checklist
- Ensure your ECS push work pool has a valid AwsCredentials block attached (not {}).
- Ensure region_name is set either in the block, the pool, or job_variables.
- If you’re overriding job_variables in the deployment, don’t pass an empty dict for `aws_credentials`; use a $ref to your block.
If you share:
- Your work pool type/name,
- A snippet of your pool’s Base job template (the variables section),
- Any job_variables in your deployment,
I can spot the exact place where {} is getting injected and suggest the minimal change.Jeandre de Bruyn
12/03/2025, 1:53 PMMarvin
12/03/2025, 1:53 PMMarvin
12/03/2025, 1:54 PMjob_variables overrides (if any).
- You only need to redeploy if you want to change something stored in the deployment itself (e.g., image, entrypoint, or job_variables defined in the deployment), or if you renamed variables and your deployment overrides no longer match.
Given you just added `aws_credentials`/`region_name` to the pool’s base job template, your existing deployments will pick that up on the next run without redeploying.
Refs:
- Serverless/push work pools: https://docs.prefect.io/v3/how-to-guides/deployment_infra/serverless
- Manage work pools and base job templates: https://docs.prefect.io/v3/how-to-guides/deployment_infra/manage-work-pools#base-job-templateJeandre de Bruyn
12/03/2025, 1:58 PMMarvin
12/03/2025, 1:58 PMMarvin
12/03/2025, 2:05 PMregion_name in the variables, and
- aws_credentials default as a block reference to aws-credentials/production-aws-credentials,
but you’re still seeing:
3 validation errors for AwsCredentials
aws_access_key_id: Field required
aws_secret_access_key: Field required
region_name: Field required
that strongly suggests Prefect is receiving {} for aws_credentials at validation time (either the block reference isn’t resolving, or something is overriding it with an empty dict), or your referenced block does not have those fields populated for your version.
Quick checks to pinpoint the issue
1) Verify the block reference resolves
- Check the block exists and the slug is correct:
prefect block inspect aws-credentials/production-aws-credentials
Confirm it shows the expected data (keys/profile and region). If it doesn’t resolve, fix the slug or recreate/select the block via the UI.
2) Prefer block_document_id over slug in the template
- Some environments only resolve block_document_id. Try replacing the slug with the exact block ID (from the UI or from prefect block inspect) in your Base job template:
"aws_credentials": {
"title": "AWS Credentials",
"type": "object",
"default": {
"$ref": {
"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
},
"description": "AWS credentials block for ECS API access"
}
Note: adding "type": "object" on the aws_credentials property is recommended.
3) Ensure nothing overrides with an empty dict
- Check your deployment(s) do not set:
work_pool.job_variables.aws_credentials: {}
in prefect.yaml or in your .deploy(..., job_variables=...).
4) Make sure the block actually has region + credentials set
- For push pools, your environment likely expects explicit values in the block. Edit the AwsCredentials block and set either:
- Access key, secret key, and region, or
- Profile name and region
If your version requires these, an empty/missing region in the block will trigger this validation.
5) Use the UI to set the block on the pool (bypasses JSON pitfalls)
- Work Pools → your ecs:push pool → Settings → AWS Credentials → select the block, and set Region if prompted. Save.
If you want, paste the outputs of:
- prefect block inspect aws-credentials/production-aws-credentials (redact secrets)
- prefect work-pool inspect "<your-pool-name>" (variables section)
- Any job_variables from your deployment
and I’ll call out the exact mismatch.
Helpful docs
- Serverless/push work pools: Serverless work pools
- Manage work pools & base job templates: Manage work pools
- AWS integration (creating AwsCredentials block): prefect-awsJeandre de Bruyn
12/03/2025, 2:14 PMMarvin
12/03/2025, 2:14 PMMarvin
12/03/2025, 2:18 PM{} for aws_credentials at validation time. That typically means either the block reference isn’t resolving or something is overriding it with an empty dict.
Let’s run through a few targeted checks/fixes.
1) Verify the pool you’re running on has the right template and reference
- Inspect the pool and look at both job_configuration and variables defaults:
prefect work-pool inspect "<your-ecs-push-pool>"
Confirm you see BOTH of these:
- job_configuration has: "aws_credentials": "{{ aws_credentials }}"
- variables has an aws_credentials default that is a $ref (not {})
If job_configuration is missing the templated line, add it. If variables.default is {}, fix it.
2) Switch from slug → block_document_id in the template
Slugs can fail to resolve in some cases. Replace your slug with the exact block ID:
- Get the ID:
prefect block inspect aws-credentials/production-aws-credentials
- Update your pool’s Base job template so the property looks like:
"aws_credentials": {
"title": "AWS Credentials",
"type": "object",
"default": {
"$ref": {
"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
},
"description": "AWS credentials block for ECS API access"
}
Then apply:
prefect work-pool update "<your-ecs-push-pool>" --base-job-template ecs_base.json
3) Ensure nothing overrides with an empty dict
- Inspect the deployment(s) for overrides:
prefect deployment inspect "<flow-name>/<deployment-name>"
If you see:
work_pool:
job_variables:
aws_credentials: {}
remove that override.
4) Quick diagnostic: set aws_credentials at the deployment level with a $ref
This bypasses the pool template’s reference resolution and can confirm the root cause:
- Using the block ID:
job_variables={
"aws_credentials": {"$ref": {"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}},
"region_name": "eu-central-1"
}
If this works, the issue is in the pool’s base job template reference.
5) As a sanity test: pass raw fields directly (no block) in job_variables
If the model accepts raw dict fields, you’ll know the validation itself is fine and the problem is block resolution:
job_variables={
"aws_credentials": {
"profile_name": "my-profile",
"region_name": "eu-central-1"
}
}
(or access key + secret + region)
6) Confirm the deployment is actually targeting this pool
It’s easy to update one pool but run the deployment on another. Double-check work_pool_name in your deployment and the pool name you inspected.
If it still errors, please paste (with secrets redacted):
- The aws_credentials portions from:
- prefect work-pool inspect "<your-ecs-push-pool>"
- prefect deployment inspect "<flow/deployment>"
- The block ID you’re referencing
I’ll pinpoint exactly where {} is sneaking in and share the minimal fix.
Helpful docs
- Serverless/push work pools: Serverless work pools
- Manage work pools & Base job templates: Manage work pools
- AWS integration (AwsCredentials block): prefect-awsJeandre de Bruyn
12/03/2025, 2:30 PMMarvin
12/03/2025, 2:30 PMMarvin
12/03/2025, 2:32 PMjob_variables: {} on the deployment is expected. Deployments only show per-deployment overrides there. If you didn’t set any overrides, job_variables will be empty and the run will inherit from the work pool’s Base job template. So that alone isn’t the problem.
Given you still get the AwsCredentials “Field required … input_value {}” error, Prefect is still receiving {} for aws_credentials at validation time. The most common causes are:
- The pool’s Base job template doesn’t wire aws_credentials into job_configuration, or
- The block reference isn’t resolving (slug issues), or
- Something else is overriding aws_credentials with {}.
Please run these checks:
1) Inspect the work pool and verify both sides are wired
- Must have the template placeholder in job_configuration:
"job_configuration": {
...
"aws_credentials": "{{ aws_credentials }}",
...
}
- And the variables default should be a block reference (ideally by ID, not slug):
"variables": {
"properties": {
"aws_credentials": {
"title": "AWS Credentials",
"type": "object",
"default": {
"$ref": {
"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
}
}
}
}
Command to inspect:
prefect work-pool inspect "production-ecs-push-pool"
2) Prefer block_document_id over slug
Some environments don’t resolve slugs the way you expect. Replace your slug with the exact block_document_id from:
prefect block inspect aws-credentials/production-aws-credentials
Then update the pool:
prefect work-pool update "production-ecs-push-pool" --base-job-template ecs_base.json
3) Verify nothing overrides with {}
Double-check the deployment doesn’t set an empty dict anywhere (even in CI templates), e.g.:
work_pool:
job_variables:
aws_credentials: {}
Your deployment shows {}, which is fine — just make sure there’s no hidden override elsewhere.
4) Quick isolation test
Set the credentials at the deployment level to bypass pool reference resolution:
- Using block_document_id
job_variables={
"aws_credentials": {"$ref": {"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}},
"region_name": "eu-central-1"
}
If that works, the issue is the pool’s template reference, not the block.
5) Version check (to rule out schema drift)
Please share:
- prefect version
- python -c "import prefect_aws, prefect; print(prefect.__version__, getattr(prefect_aws, '__version__', 'unknown'))"
If you can paste the aws_credentials-related parts of:
- prefect work-pool inspect "production-ecs-push-pool" (job_configuration and variables for aws_credentials)
- prefect block inspect aws-credentials/production-aws-credentials (redact secrets)
I’ll point to the exact mismatch and the minimal change needed.
Helpful docs:
- Serverless/push work pools: Serverless work pools
- Manage work pools / Base job template: Manage work pools
- AWS integration and AwsCredentials block: prefect-awsJeandre de Bruyn
12/03/2025, 2:41 PMMarvin
12/03/2025, 2:41 PMMarvin
12/03/2025, 2:45 PMtask_definition dict (so Prefect can register one), or
- a task_run_request.taskDefinition value (usually a task definition ARN).
- Your template likely has the default task_run_request.taskDefinition: "{{ task_definition_arn }}" and you didn’t set task_definition_arn in job variables, and you also did not provide a task_definition dict. So both are empty → validation fails.
Two ways to fix (pick one)
Option A: Use an existing task definition ARN
- In your work pool’s Base job template, ensure this is present:
"job_configuration": {
...
"task_run_request": {
...
"taskDefinition": "{{ task_definition_arn }}",
...
}
}
- Then, set task_definition_arn somewhere:
- Per-deployment override:
- Python .deploy(...):
job_variables={
"task_definition_arn": "arn:aws:ecs:eu-central-1:123456789012:task-definition/my-task:5"
}
- Or in prefect.yaml:
work_pool:
job_variables:
task_definition_arn: "arn:aws:ecs:eu-central-1:123456789012:task-definition/my-task:5"
- Or embed a default in the pool variables:
"variables": {
"properties": {
"task_definition_arn": {
"type": "string",
"default": "arn:aws:ecs:eu-central-1:123456789012:task-definition/my-task:5"
}
}
}
- This satisfies the validator via task_run_request.taskDefinition.
Option B: Let Prefect register a task definition for you
- Provide a task_definition dict in the pool’s job_configuration that Prefect can register. Minimal example:
"job_configuration": {
...
"task_definition": {
"family": "{{ family }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"executionRoleArn": "{{ execution_role_arn }}",
"containerDefinitions": [
{
"name": "{{ container_name }}",
"image": "{{ image }}",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/prefect/flows",
"awslogs-region": "{{ region_name }}",
"awslogs-stream-prefix": "prefect"
}
}
}
]
},
"task_run_request": {
"cluster": "{{ cluster }}",
"launchType": "{{ launch_type }}",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": "{{ subnets }}",
"securityGroups": "{{ security_groups }}",
"assignPublicIp": "{{ assign_public_ip }}"
}
}
}
}
- Ensure the variables referenced (image, family, cpu, memory, execution_role_arn, region_name, subnets, security_groups, assign_public_ip, etc.) have sensible defaults in variables.properties.
Which should you choose?
- If you already manage task definitions (IaC, console), Option A is simplest: set task_definition_arn.
- If you want Prefect to handle task definition creation/update per run, use Option B.
Next steps for you
- Decide A or B above.
- If A: set task_definition_arn at the deployment or pool level and retry a run.
- If B: add a task_definition block to job_configuration and ensure the referenced variables are present.
If you share your current job_configuration section and the variables you have defined (redact secrets), I can propose the minimal JSON patch to get you unblocked.Jeandre de Bruyn
12/03/2025, 2:49 PMMarvin
12/03/2025, 2:49 PMMarvin
12/03/2025, 2:50 PM{
"job_configuration": {
"aws_credentials": "{{ aws_credentials }}",
"task_run_request": {
"cluster": "arn:aws:ecs:eu-central-1:...:cluster/floryn-prediction-services",
"launchType": "FARGATE",
"taskDefinition": "{{ task_definition_arn }}",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-x"
],
"securityGroups": [
"sg-x"
],
"assignPublicIp": "DISABLED"
}
},
"overrides": {
"containerOverrides": [
{
"name": "prefect",
"command": "{{ command }}",
"environment": [
{
"name": "PREFECT_API_URL",
"value": "<https://api.prefect.cloud/api/x>"
}
],
"cpu": "{{ cpu }}",
"memory": "{{ memory }}"
}
],
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"taskRoleArn": "{{ task_role_arn }}"
},
"tags": "{{ labels }}"
}
},
"variables": {
"properties": {
"task_definition_arn": {
"type": "string",
"default": "arn:aws:ecs:eu-central-1:...:task-definition/my-task:5"
},
"cpu": {
"type": "integer",
"default": 16384
},
"memory": {
"type": "integer",
"default": 65536
},
"task_role_arn": {
"type": "string",
"default": "arn:aws:iam::...:role/prefect-task-role"
},
"image": {
"type": "string"
},
"region_name": {
"type": "string",
"default": "eu-central-1"
},
"aws_credentials": {
"default": {
"$ref": {
"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
}
}
}
}
}
Option B — Let Prefect register a task definition for you
Provide a proper task_definition (camelCase) and keep task_run_request without a taskDefinition:
```
{
"job_configuration": {
"aws_credentials": "{{ aws_credentials }}",
"task_definition": {
"family": "{{ family }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"executionRoleArn": "{{ execution_role_arn }}",
"containerDefinitions": [
{
"name": "prefect",
"image": "{{ image }}",
"essential": true,
"environment": [
{"name": "PREFECT_API_URL", "value": "https://api.prefect.cloud/api/x"}
],
"secrets": [
{"name": "PREFECT_API_KEY", "valueFrom": "arnawssecretsmanager...secret:..."}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/prefect/flows",
"awslogs-region": "{{ region_name }}",
"awslogs-stream-prefix": "prefect"
}
}
}
]
},
"task_run_request": {
"cluster": "arnawsecseu central 1...:cluster/floryn-prediction-services",
"launchType": "FARGATE",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": ["subnet-x"],
"securityGroups": ["sg-x"],Marvin
12/03/2025, 2:50 PMJeandre de Bruyn
12/03/2025, 2:51 PMMarvin
12/03/2025, 2:51 PMMarvin
12/03/2025, 2:53 PMjob_configuration with the nested/camelCase structure below.
Drop this into your Base job template and update the pool. It uses your existing values and adds the minimal required ECS fields.
{
"job_configuration": {
"aws_credentials": "{{ aws_credentials }}",
"task_definition": {
"family": "{{ family }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"networkMode": "awsvpc",
"requiresCompatibilities": [
"FARGATE"
],
"executionRoleArn": "{{ execution_role_arn }}",
"containerDefinitions": [
{
"name": "prefect",
"image": "{{ image }}",
"essential": true,
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"environment": [
{
"name": "PREFECT_API_URL",
"value": "<https://api.prefect.cloud/api/x>"
}
],
"secrets": [
{
"name": "PREFECT_API_KEY",
"valueFrom": "arn:aws:secretsmanager:eu-central-1:ACCOUNT:secret:SECRET-NAME"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/prefect/flows",
"awslogs-region": "{{ region_name }}",
"awslogs-stream-prefix": "prefect"
}
}
}
]
},
"task_run_request": {
"cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/floryn-prediction-services",
"launchType": "FARGATE",
"overrides": {
"containerOverrides": [
{
"name": "prefect",
"command": "{{ command }}",
"environment": "{{ env }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}"
}
],
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"taskRoleArn": "{{ task_role_arn }}"
},
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-x"
],
"securityGroups": [
"sg-x"
],
"assignPublicIp": "DISABLED"
}
},
"tags": "{{ labels }}"
}
},
"variables": {
"type": "object",
"properties": {
"aws_credentials": {
"title": "AWS Credentials",
"type": "object",
"default": {
"$ref": {
"block_document_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
}
},
"region_name": {
"type": "string",
"default": "eu-central-1"
},
"family": {
"type": "string",
"default": "prefect-flows"
},
"image": {
"type": "string"
},
"container_name": {
"type": "string",
"default": "prefect"
},
"cpu": {
"type": "integer",
"default": 16384
},
"memory": {
"type": "integer",
"default": 65536
},
"cluster": {
"type": "string",
"default": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/floryn-prediction-services"
},
"launch_type": {
"type": "string",
"default": "FARGATE"
},
"execution_role_arn": {
"type": "string",
"default": "arn:aws:iam::ACCOUNT:role/prefect-task-execution-role"
},
"task_role_arn": {
"type": "string",
"default": "arn:aws:iam::ACCOUNT:role/prefect-task-role"
},
"subnets": {
"type": "array",
"items": {
"type": "string"
},
"default": [
"subnet-x"
]
},
"security_groups": {
"type": "array",
"items": {
"type": "string"
},
"default": [
"sg-x"
]
},
"assign_public_ip": {
"type": "string",
"default": "DISABLED"
},
"env": {
"type": "object",
"default": {},
"additionalProperties": {
"type": "string"
}
},
"labels": {
"type": "object",
"default": {},
"additionalProperties": {
"type": "string"
}
},
"command": {
"type": "array",
"items": {
"type": "string"
},
"default": []
}
}
}
}
Notes
- Keys must be camelCase in task_definition and task_run_request (e.g., containerDefinitions, awsvpcConfiguration, assignPublicIp).Marvin
12/03/2025, 2:53 PMnetworkMode: "awsvpc" and requiresCompatibilities: ["FARGATE"] in task_definition.
- Remove the old flat/snake_case keys (container_definitions, network_configuration, region).
- Ensure your secrets ARN is correct.
- Your cpu/memory pair (16384/65536) is valid for Fargate in eu-central-1.
Apply and test
- Save JSON as ecs_base.json, then:
prefect work-pool update "production-ecs-push-pool" --base-job-template ecs_base.json
- No redeploy needed — try triggering a run again.
If it errors again, paste the updated prefect work-pool inspect "production-ecs-push-pool" job_configuration section and we’ll iterate. Helpful refs:
- Manage work pools / Base job template: Manage work pools
- Serverless/push work pools overview: Serverless work pools