I m trying to set up S3 storage for flow runs using prefect

Question

I m trying to set up S3 storage for flow runs using prefect 1 ```with Flow test flow storage=S3 bucket= my bucket run config=ECSRun ``` but getting this error ```Failed to load and execute flow run NoCredentialsError Unable to locate credentials ``` I added a Secret named `AWS CREDENTIALS` through the prefect cloud UI <https cloud prefect io team secrets> with a json of the form ``` ACCESS KEY SECRET ACCESS KEY ``` but still getting this error Curious if anyone one has tips to troubleshoot

Nate · Answer

Hi < Leo Marvin> when do you get this error trying to run the flow locally after `prefect register ` sometime else

Leo Marvin · Answer

Triggering the flow in the UI i e quick run Or triggering locally ```prefect run ```

Leo Marvin · Answer

```Downloading flow from s3 Error downloading Flow from S3 Unable to locate credentials Failed to load and execute flow run NoCredentialsError Unable to locate credentials ```

Leo Marvin · Answer

Other context is that credentials have a very limited set of access just GetObject on the bucket for now not sure if I need to add more access but the error seems to indicate to me that prefect isn t finding any creds vs not having enough access so haven t gone down the path of expanding access I wasn t sure if I configured the secret wrong or needed to register it to the flow in some way

Nate · Answer

have you seen you could populate your `default` aws profile with your creds or create another profile if you have multiple like me and the right one and then when you run `prefect agent ecs start ` they should get picked up and used by the agent I don t believe you d need to set them as a secret unless you needed their values in the context of a flow run for some reason EDIT for running in production you can define an Execution Role in AWS IAM and then reference its ARN when you do `prefect agent ecs start execution role your arn`

Leo Marvin · Answer

Oh I didn t run ```prefect agent ecs start``` I ran ```aws ecs create service ``` which created my ECS agent It does appear that starting the ECS agent myself with `prefect agent start` seems to work though

Leo Marvin · Answer

< Nate> Sorry still confused here gt for running in production you can define an Execution Role in AWS IAM and then reference its ARN when you do `prefect agent ecs start execution role your arn` My understanding from is that `prefect agent start` is not how you run in production You instead create a service which is what I did originally with `aws ecs create service` The fact that this works with `prefect agent start` doesn t seem to solve my original question of wanting to deploy this in production Am I missing something

Prefect · Answer

hey < Leo Marvin> Even when you use the aws CLI to register the task definition JSON inside you ll see that it s using `prefect agent ecs start` under the hood I think the creation of that service from the task definition JSON is the recommendation for production because it s more like infra as code than a one off CLI command Apologies that I m currently on mobile but I will circle back to this later and see if you still need help

Leo Marvin · Answer

Ah adding `task definition arn` to my `run config` `ECSRun ` command solved all my issues Thank you so much for the help