Hello all, I am running a prefect ECS agent as a service in AWS ECS. Now I am trying to register a flow run in Prefect Cloud using the Docker storage. I created a

ECSRun

run configuration for the

run_config

argument in the Flow. If I use the

env

argument to provide the

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

it works! However, I don’t want to put my credentials in the source code. I am trying to use the

task_definition_arn

argument, but I am not sure if it is the correct way, because the image of the task should be the Flow docker image. Can you give me any tip or advise? Thank you very much!

Hi @jcozar are you setting AWS credentials on your ECSAgent? I may be missing something but I’m not sure how setting them on the run config would effect the agent being able to create the task 🤔

Thank you very much for your fast response! In order to simplify the problem, let’s assume that I am using AWS S3 Storage. The prefect ecs agent is running in ECS using a custom task definition. The task role has enough privileges to access S3 and to run new tasks in ECS. I register a new Flow in Prefect Cloud using a ECSRun Run Configuration. When I run a new flow run manually, the task in my ECS cluster appears. However: • If I don’t provide the access and secret access key to the

RunConfig

, then, the new task raises the exception

NoCredentialsError('Unable to locate credentials')

when trying to download the flow from S3 (last info log message is

Downloading flow from s3://...

• If I provide the access and secret access key to the

RunConfig

, it has permissions to download the flow and execute the flow run. I read in the documentation that if the

task_definition

is not specified in the

RunConfig

, the default configuration for the task_definitions is used (one per each flow run version). The default configuration uses None for the task_execution_role, and that’s why it cannot download the flow definition from S3. The same issue if I use Docker and AWS ECR registry. I tried to use the

execution_role_arn

to use a full privileged role, but the problem is not creating the task, is trying to download the flow (storage) from S3 in runtime. I did’t try the combination of the Docker Storage and the

execution_role_arn

with full privileged (maybe in this case the task image is the one from AWS ECR). I’m gonna try it!

Ah yeah the

task_role_arn

(either set as default on the agent or on the flow’s run config) is what allows the flow to use the S3 storage inside the job kicked off by the agent. This is how you avoid having to set the access keys directly in the run config. I also think that the

execution_role_arn

is what is needed to pull the ECR image however that wouldn’t be a responsibility of the job itself and instead would be something the agent would need (or if you provide access keys to the agent it would accomplish the same behavior)

Ok thank you! Indeed, if I use the Docker storage and provide the

excution_role_arn

then the flow run is executed correctly! However, in the case that I need to use S3 storage, how can I provide the

task_role_arn

? The only way I see is to specify a task definition, which makes sense to me 🙂 Is this right? Thank you!

The flow’s ECSRun allows for setting the

task_role_arn

https://docs.prefect.io/api/latest/run_configs.html#ecsrun

Thank you! I read all the arguments but I missread that one!