Hi Everyone, we seem to be experiencing an unexpected issue with one of our ECS agents. When kicking off a flow, we get an error message:

Unable to locate credentials

. This happens right after the flow is scheduled and submitted (before any of the tasks can even begin to kick off). We don't think it is related to boto because our EC2 instance is able to read from aws buckets (confirmed by ssh'ing on the instance) via an iam policy attached to the EC2 instance. Unfortunately, running with debug mode didn't produce any more verbose results or a stack trace. We did have to reboot this instance recently after it became unresponsive due to an errant task we had running that consumed too much memory and stalled out. Has anyone experienced this before?

Hey @Danielle Dalton , weird it stopped working if this was working before. What flow storage are you using? S3? ECR? Github?

We use S3 as our flow storage.

You had this working without supplying credentials before just by using the EC2 role?

Yes, we created an iam instance profile that specified access to our S3 resources and then we assigned the iam instance profile to the EC2 instance.

I don’t have much advice here. It seems right. Unfortunately, ECS is a massive pain when things don’t start cuz there are no logs anywhere. The only thing I can suggest you try is explicitly passing AWS credentials to see if that works. Will ask another team member if there are better ideas.

Not a problem - thanks for the suggestion Kevin. When we figure this out, we'll share our answer here for others in the future.

Do you have a

task_role_arn

on the ECS agent?

Yes we do. No, we don't have a task role arn on the agent because we only use the agent for launching pre-existing ECS tasks.

Hey @Kevin Kho, we "fixed" this by restarting the systemd services that keep our prefect agents up and running. I'm wondering if this was related to the fact that we had to reboot our EC2 instance, and perhaps by some race condition the systemd service brought the ECSAgent back up before the instance was aware of the fact that it could authenticate to AWS via an instance profile

Gotcha so we normally recommend using an ECSService instead of EC2 for spinning up the Agent. I’ll keep this in mind though in case someone runs into the same issue. That race condition sounds weird, but I totally believe you.

interesting okay, I can see that a lot of this documentation was published a little while after we got our ECSAgents up and running in production; I'll take a look at this!