https://prefect.io logo
Title
l

Leo Marvin

08/01/2022, 8:04 PM
I'm trying to set up S3 storage for flow runs (using prefect 1)
with Flow(
    "test-flow",
    storage=S3(bucket="my-bucket"),
    run_config=ECSRun(...)
but getting this error:
Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
I added a Secret named
AWS_CREDENTIALS
through the prefect cloud UI (https://cloud.prefect.io/team/secrets) with a json of the form
{
  "ACCESS_KEY": "...",
  "SECRET_ACCESS_KEY": "..."
}
but still getting this error. Curious if anyone one has tips to troubleshoot.
āœ… 1
n

Nate

08/01/2022, 8:11 PM
Hi @Leo Marvin! when do you get this error? ā€¢ trying to run the flow locally? ā€¢ after
prefect register ..
ā€¢ sometime else?
l

Leo Marvin

08/01/2022, 8:16 PM
Triggering the flow in the UI i.e. "quick-run" Or triggering locally
prefect run ...
Downloading flow from s3://...
Error downloading Flow from S3: Unable to locate credentials
Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
Other context is that credentials have a very limited set of access (just GetObject on the bucket for now), not sure if I need to add more access but the error seems to indicate to me that prefect isn't finding any creds vs not having enough access so haven't gone down the path of expanding access. I wasn't sure if I configured the secret wrong or needed to register it to the flow in some way.
n

Nate

08/01/2022, 8:32 PM
have you seen these docs? you could populate your
default
aws profile with your creds (or create another profile if you have multiple like me and awsume the right one) and then when you run
prefect agent ecs start ...
they should get picked up and used by the agent I don't believe you'd need to set them as a secret, unless you needed their values in the context of a flow run for some reason EDIT: for running in production, you can define an Execution Role in AWS IAM and then reference its ARN when you do
prefect agent ecs start --execution-role your:arn
l

Leo Marvin

08/01/2022, 10:29 PM
Oh I didn't run
prefect agent ecs start
I ran
aws ecs create-service \
  ...
which created my ECS agent. It does appear that starting the ECS agent myself with
prefect agent start
seems to work though.
šŸ‘ 1
@Nate Sorry still confused here.
for running in production, you can define an Execution Role in AWS IAM and then reference its ARN when you do
prefect agent ecs start --execution-role your:arn
My understanding from the docs is that
prefect agent start
is not how you run in production. You instead create a service which is what I did originally with
aws ecs create-service
. The fact that this works with
prefect agent start
doesn't seem to solve my original question of wanting to deploy this in production. Am I missing something?
p

Prefect

08/01/2022, 11:12 PM
hey @Leo Marvin Even when you use the aws CLI to register the task definition JSON, inside you'll see that it's using
prefect agent ecs start
under the hood, I think the creation of that service from the task definition JSON is the recommendation for production because it's more like infra as code than a one off CLI command. Apologies that I'm currently on mobile, but I will circle back to this later and see if you still need help
l

Leo Marvin

08/01/2022, 11:29 PM
Ah, adding
task_definition_arn
to my
run_config
ECSRun()
command solved all my issues. Thank you so much for the help.
šŸ‘ 1
šŸš€ 1
:marvin-duck: 1