https://prefect.io logo
Join Slack
Powered by
Hello, running a worker locally or in ECS, I getti...
# ask-community
b
Hello, running a worker locally or in ECS, I getting this error when submitting a job to ECS:
Copy code
Flow run could not be submitted to infrastructure:
ProfileNotFound('The config profile (prefect) could not be found')
No matter what I do to ensure the AWS credentials, I am always getting this error.
Its coming from the boto library:
Copy code
File "/usr/local/lib/python3.12/site-packages/botocore/session.py", line 424, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (prefect) could not be found
b
Do you have 
AWS_DEFAULT_PROFILE
or 
AWS_PROFILE
set in your environment variables somewhere?
Something is telling 
botocore
to look for a profile called 
prefect
in your 
~/.aws/config
file
b
Hey @Brendan Dalpe - yeah I have been trying to figure why its looking for this profile. At this point I have run the worker command locally (with my own AWS credentials) on AWS ECS which has an IAM Role configured, and finally within a docker compose setup. I've tried to hard code the access and secret keys directly in the environment but it still tried to find this profile
b
How'd you deploy your ECS worker?
b
👍 1
I can send the task config if that helps, the only changes from the guide are adding 
AWS_ACCESS_KEY_ID
and 
AWS_SECRET_ACCESS_KEY
environment vars. Using image 
prefecthq/prefect:3-latest
and start up command:
Copy code
"command": [
        "/bin/sh",
        "-c",
        "pip install prefect-aws && prefect worker start --pool aws-ecs-work-pool --type ecs"
      ],
To be a little more clear maybe... the worker service is up and running correctly, its able to pick up jobs from the Prefect server.... but this error occurs when trying to submit a job to ECS for execution
b
Your worker shouldn't need the static keys. Following the guide, the worker has an ECS task IAM role assigned to it which it will use for creating other ECS tasks when running Prefect flows.
👍 1
b
Hey yeah I know, I only hard coded the keys to test. But even with the keys directly in the env, it was looking for this profile
👍 1
b
More than likely, it wants a default region and it's trying to find that in the profile. Can you try setting 
AWS_REGION=<whatever your region is>
and 
AWS_DEFAULT_REGION
and see if that helps?
b
I have 
AWS_DEFAULT_REGION
set already (forget to mention that)
👍 1
I know its a bit much to ask, but just wondering if you have the ability to go through the ECS guide and try it? Its driving me insane 😆
because I have now tried in 3 different environments and getting the same error.... I have a feeling its a bug in the code, what else could explain it? And the fact that its specifically looking for a profile named 
prefect
, where is it getting that?
b
Agreed. I'm unable to locate the 
prefect
profile thing which also has me scratching my head.
I'll run through the guide in my lab and see what I come up with.
In the mean time, how's your Terraform-fu? We do have a module for this that you can feel free to try out and see if you get any different results: https://github.com/PrefectHQ/terraform-prefect-ecs-worker
b
oh wow, yeah im using terraform heavily, I will check it out
🔥 1
Thank you so much Brendan, I really appreciate you taking the time. If you discover anything or need anything, please let me know
b
So, I ran through the guide without adding the environment variables and I'm not able to run a task because I get an error:
Copy code
AccessDeniedException('An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation: User: arn:aws:sts::xxxx:assumed-role/xxxxxxx is not authorized to perform: ecs:RegisterTaskDefinition on resource: arn:aws:ecs:us-east-2:xxxx:task-definition/prefect_my-ecs-pool_73c9ffab-a5f9-4aa6-aef8-ea736f8c16bf:* because no identity-based policy allows the ecs:RegisterTaskDefinition action')
I'll get the relevant permissions/steps added to the docs.
I'm now testing with the explicit credentials as environment variables like you were trying.
Ok, that yields the same result...
b
Is your first error is related to the assume role trust policy?
b
Yep, I don't get the profile error like you were seeing.
Can you potentially DM me your command history in a markdown/txt doc? Obfuscate the connection strings please
b
yeah I can try, I was doing a bit of click-formation in AWS for all the roles and policies
thank you 1
👀 1
but I think your error is actually related to the iam role not being correct configured. particularly steps 1 and 2 -> https://docs.prefect.io/integrations/prefect-aws/ecs_guide#1-create-a-trust-policy
b
Yes, this is a different issue I need to address in the docs.
Even if you get all of the steps right, you're still missing permissions to actually register tasks in ECS which will cause the worker to be unable to dispatch workflows. This will show them as crashed in Prefect.
Public update: The profile name was set in the AWS Credentials block, deleting fixed the original issue.
b
Yep, when configuring the AWS Credentials block in Prefect, I had set the "Profile Name" which was causing the worker to search for a profile named 
prefect
6 Views
Open in Slack
PreviousNext
Powered by