Hello I am running Prefect self hosted on an EC2 instance in Prefect Community #ask-community

Hello, I am running Prefect self hosted on an EC2 ...

Ben

08/05/2025, 3:59 PM

Hello, I am running Prefect self hosted on an EC2 instance in AWS. I have configured an AWS ECS work pool and am following this guide to create an AWS ECS based worker to pick up jobs. The worker is correctly registered with my Prefect server and the work pool is active. The start up command is:

Copy code

"command": [
        "/bin/sh",
        "-c",
        "pip install prefect-aws && prefect worker start --pool aws-ecs-work-pool --type ecs"
      ],

When I execute a deployment, the job is successfully picked up by the worker, however it immediately fails when trying to submit the job to ECS for task execution. I get this error:

Copy code

File "/usr/local/lib/python3.12/site-packages/botocore/session.py", line 424, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (prefect) could not be found

prefect.flow_runs.worker
Reported flow run '4ab3b0a2-5e29-4031-a621-f7159b46a880' as crashed: Flow run could not be submitted to infrastructure:
ProfileNotFound('The config profile (prefect) could not be found')

The ECS worker has an IAM role attached to it using the

taskRoleArn

and

executionRoleArn

configurations in the task definition I have even tried hard coding the

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

environment vars in the task configuration. But no matter what I do, the execution always fails immediately with this error. Its like its ignoring all of my AWS credentials config and always trying to find a profile with the name

prefect

which doesnt exist and fails. Please help!

Ben

08/05/2025, 4:24 PM

I tried to run the worker in a docker compose environment from the same host running the Prefect server. The dockerfile looks like this:

Copy code

# Dockerfile
FROM prefecthq/prefect:3-latest

RUN pip install -U "prefect[gcp,aws]"

and the start command is:

Copy code

entrypoint:
      - /opt/prefect/entrypoint.sh
      - prefect
      - worker
      - start
      - '--pool=aws-ecs-work-pool'
      - '--with-healthcheck'

The env vars are like:

Copy code

environment:
      - 'PREFECT_API_URL=<https://prefect.example.com/api>'
      - 'PREFECT_API_KEY=${SERVICE_PASSWORD_APIKEY}'
      - 'DEFAULT_POOL_NAME=${DEFAULT_POOL_NAME:-default}'
      - 'AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}'
      - 'AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}'
      - 'AWS_REGION=${AWS_REGION:-us-west-2}'

So the AWS credentials are included in the runtime environment as when it successfully picks up the task.... it fails with the same error:

Copy code

16:19:01.204 | INFO    | prefect.flow_runs.worker - Reported flow run 'f6740f0f-e635-4017-a73c-c8907010f99e' as crashed: Flow run could not be submitted to infrastructure:
agent-aws-esc-1  | ProfileNotFound('The config profile (prefect) could not be found')

Ben

08/05/2025, 4:58 PM

Another update: I get the same error when I run the worker command locally as well:

Copy code

prefect worker start --pool "aws-ecs-work-pool"

Before running that command, I run

aws sts get-caller-identity

to confirm I was connected to AWS

Ben

08/05/2025, 4:59 PM

The docker and ECS envs are running the lastest Prefect docker image and locally I am running

prefect==3.4.9

34 Views

Open in Slack

Previous Next