Leo Marvin
08/01/2022, 8:04 PMwith Flow(
"test-flow",
storage=S3(bucket="my-bucket"),
run_config=ECSRun(...)
but getting this error:
Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
I added a Secret named AWS_CREDENTIALS
through the prefect cloud UI (https://cloud.prefect.io/team/secrets) with a json of the form
{
"ACCESS_KEY": "...",
"SECRET_ACCESS_KEY": "..."
}
but still getting this error. Curious if anyone one has tips to troubleshoot.Nate
08/01/2022, 8:11 PMprefect register ..
⢠sometime else?Leo Marvin
08/01/2022, 8:16 PMprefect run ...
Downloading flow from s3://...
Error downloading Flow from S3: Unable to locate credentials
Failed to load and execute flow run: NoCredentialsError('Unable to locate credentials')
Nate
08/01/2022, 8:32 PMdefault
aws profile with your creds (or create another profile if you have multiple like me and awsume the right one) and then when you run prefect agent ecs start ...
they should get picked up and used by the agent
I don't believe you'd need to set them as a secret, unless you needed their values in the context of a flow run for some reason
EDIT: for running in production, you can define an Execution Role in AWS IAM and then reference its ARN when you do prefect agent ecs start --execution-role your:arn
Leo Marvin
08/01/2022, 10:29 PMprefect agent ecs start
I ran
aws ecs create-service \
...
which created my ECS agent.
It does appear that starting the ECS agent myself with prefect agent start
seems to work though.for running in production, you can define an Execution Role in AWS IAM and then reference its ARN when you doMy understanding from the docs is thatprefect agent ecs start --execution-role your:arn
prefect agent start
is not how you run in production. You instead create a service which is what I did originally with aws ecs create-service
. The fact that this works with prefect agent start
doesn't seem to solve my original question of wanting to deploy this in production. Am I missing something?Prefect
08/01/2022, 11:12 PMprefect agent ecs start
under the hood, I think the creation of that service from the task definition JSON is the recommendation for production because it's more like infra as code than a one off CLI command.
Apologies that I'm currently on mobile, but I will circle back to this later and see if you still need helpLeo Marvin
08/01/2022, 11:29 PMtask_definition_arn
to my run_config
ECSRun()
command solved all my issues.
Thank you so much for the help.