@Marvin Why am I getting this error when trying to run a flow with an ECS Push queue?

Flow run could not be submitted to infrastructure: The config profile (default) could not be found

Oh, the joys of error messages. They're like cryptic little puzzles, aren't they? Your error message suggests that the ECS agent is unable to find the default configuration profile. This could be due to a few reasons: 1. Your AWS credentials are not properly set up. Make sure your AWS credentials are correctly configured in your environment. You can do this by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, or by using the AWS CLI's

aws configure

command. 2. The ECS agent is not properly configured. Ensure that the agent has the necessary permissions and access to pull your flow run’s image and spin up the required resources. 3. There might be an issue with your Prefect configuration. You might want to check your Prefect configuration file to ensure that it's correctly set up. 4. The agent might be polling too frequently and not having enough resources to deploy runs to infrastructure. You could try decreasing the poll frequency. 5. There could be more than one agent polling for runs from the same work queue. This can often lead to some runs that can’t get deployed efficiently. Remember, the universe is vast and complicated, and sometimes things just don't work the way we want them to. But don't worry, with a little bit of patience and debugging, you'll get there.

@Marvin Is there a way to validate that the AWS Credentials block is working correctly?

Ah, validation. It's like asking the universe for a sign that you're on the right path. Unfortunately, the universe rarely obliges. But in the case of AWS credentials, there are a few ways to validate that they're working correctly: 1. You can use the AWS CLI to test your credentials. Run a simple command like

aws s3 ls

to list your S3 buckets. If your credentials are set up correctly, you should see a list of your buckets. If not, you'll get an error message. 2. You can also use the AWS SDKs to test your credentials. For example, in Python, you can use the

boto3

library to interact with AWS services. Try to create a client for a service and make a request. If your credentials are valid, the request will succeed. If not, you'll get an error. 3. If you're using an IAM role, you can check the role's access in the AWS Management Console. Navigate to the IAM service, find your role, and check its access policies. 4. If you're using ECS, you can check the ECS agent logs for any error messages related to authentication. Remember, when it comes to validation, the universe may be silent, but your computer is not. Listen to the error messages, they're trying to tell you something.

you could use them list your bucket to see if they work

They work fine locally, but don't seem to be working with Prefect

locally boto3 will pull your creds from ~/.aws/credentials, do you have access key id and secret in your aws creds block?

I'm using a profile and SSO in the config file instead of the standard named profile method. Boto3 happily uses this method.

hmm do you have the full trace from when it failed?

Yep, here you go:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 521, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 618, in run
    boto_session, ecs_client = await run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 733, in _get_session_and_client
    boto_session = self.aws_credentials.get_boto3_session()
  File "/usr/local/lib/python3.9/site-packages/prefect_aws/credentials.py", line 99, in get_boto3_session
    return boto3.Session(
  File "/usr/local/lib/python3.9/site-packages/boto3/session.py", line 90, in __init__
    self._setup_loader()
  File "/usr/local/lib/python3.9/site-packages/boto3/session.py", line 131, in _setup_loader
    self._loader = self._session.get_component('data_loader')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 802, in get_component
    return self._components.get_component(name)
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 1140, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 199, in <lambda>
    lambda: create_loader(self.get_config_variable('data_path')),
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 323, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 459, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 665, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 755, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 422, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

hmm - yeah that doesnt mean anything immediately to me. one more quick thought, on the machine where that ran , whats your pydantic version?

I've got pydantic 2.4.2 and pydantic_core 2.10.1

got it thanks. yeah my suspicion just based on this

botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

is that one way or another, something must be weird with this

I'm using a profile and SSO in the config file instead of the standard named profile method

is it possible for you to

aws s3 ls

or something where the profile youre attempting to use is set up?

If I run the aws command on my local CLI it works as expected. I have tried making it a normal profile too and that works fine locally too. If it's an ECS Push work pool, do I need a worker? I assumed Prefect Cloud was taking that role?

facepalm ah man im sorry, i missed the bit about the push work pool - sorry for the birdwalk you're correct, you do not need to run a worker if using one. i'd guess its an issue with the credentials block values themselves. will look more closely at that tomorrow!

OK thanks Nate, I appreciate it. If there's any logs or something that I can get together to help, let me know 🙂

what

image

are you providing?

I've tried not putting one in (which I assume default to the Prefect one), tried putting in the Prefect one (

prefecthq/prefect:2-python3.10

) and tried my own image on AWS ECR