jcozar
01/27/2021, 5:01 PMECSRun
run configuration for the run_config
argument in the Flow. If I use the env
argument to provide the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
it works! However, I don’t want to put my credentials in the source code. I am trying to use the task_definition_arn
argument, but I am not sure if it is the correct way, because the image of the task should be the Flow docker image. Can you give me any tip or advise? Thank you very much!josh
01/27/2021, 5:30 PMjcozar
01/27/2021, 5:46 PMRunConfig
, then, the new task raises the exception NoCredentialsError('Unable to locate credentials')
when trying to download the flow from S3 (last info log message is Downloading flow from s3://...
• If I provide the access and secret access key to the RunConfig
, it has permissions to download the flow and execute the flow run.
I read in the documentation that if the task_definition
is not specified in the RunConfig
, the default configuration for the task_definitions is used (one per each flow run version). The default configuration uses None for the task_execution_role, and that’s why it cannot download the flow definition from S3. The same issue if I use Docker and AWS ECR registry.
I tried to use the execution_role_arn
to use a full privileged role, but the problem is not creating the task, is trying to download the flow (storage) from S3 in runtime. I did’t try the combination of the Docker Storage and the execution_role_arn
with full privileged (maybe in this case the task image is the one from AWS ECR). I’m gonna try it!josh
01/27/2021, 6:09 PMtask_role_arn
(either set as default on the agent or on the flow’s run config) is what allows the flow to use the S3 storage inside the job kicked off by the agent. This is how you avoid having to set the access keys directly in the run config.
I also think that the execution_role_arn
is what is needed to pull the ECR image however that wouldn’t be a responsibility of the job itself and instead would be something the agent would need (or if you provide access keys to the agent it would accomplish the same behavior)jcozar
01/27/2021, 6:13 PMexcution_role_arn
then the flow run is executed correctly! However, in the case that I need to use S3 storage, how can I provide the task_role_arn
? The only way I see is to specify a task definition, which makes sense to me 🙂 Is this right? Thank you!josh
01/27/2021, 6:14 PMtask_role_arn
https://docs.prefect.io/api/latest/run_configs.html#ecsrunjcozar
01/27/2021, 6:18 PM