Hello all, I am running a prefect ECS agent as a s...
# prefect-community
Hello all, I am running a prefect ECS agent as a service in AWS ECS. Now I am trying to register a flow run in Prefect Cloud using the Docker storage. I created a
run configuration for the
argument in the Flow. If I use the
argument to provide the
it works! However, I don’t want to put my credentials in the source code. I am trying to use the
argument, but I am not sure if it is the correct way, because the image of the task should be the Flow docker image. Can you give me any tip or advise? Thank you very much!
Hi @jcozar are you setting AWS credentials on your ECSAgent? I may be missing something but I’m not sure how setting them on the run config would effect the agent being able to create the task 🤔
Thank you very much for your fast response! In order to simplify the problem, let’s assume that I am using AWS S3 Storage. The prefect ecs agent is running in ECS using a custom task definition. The task role has enough privileges to access S3 and to run new tasks in ECS. I register a new Flow in Prefect Cloud using a ECSRun Run Configuration. When I run a new flow run manually, the task in my ECS cluster appears. However: • If I don’t provide the access and secret access key to the
, then, the new task raises the exception
NoCredentialsError('Unable to locate credentials')
when trying to download the flow from S3 (last info log message is
Downloading flow from s3://...
• If I provide the access and secret access key to the
, it has permissions to download the flow and execute the flow run. I read in the documentation that if the
is not specified in the
, the default configuration for the task_definitions is used (one per each flow run version). The default configuration uses None for the task_execution_role, and that’s why it cannot download the flow definition from S3. The same issue if I use Docker and AWS ECR registry. I tried to use the
to use a full privileged role, but the problem is not creating the task, is trying to download the flow (storage) from S3 in runtime. I did’t try the combination of the Docker Storage and the
with full privileged (maybe in this case the task image is the one from AWS ECR). I’m gonna try it!
Ah yeah the
(either set as default on the agent or on the flow’s run config) is what allows the flow to use the S3 storage inside the job kicked off by the agent. This is how you avoid having to set the access keys directly in the run config. I also think that the
is what is needed to pull the ECR image however that wouldn’t be a responsibility of the job itself and instead would be something the agent would need (or if you provide access keys to the agent it would accomplish the same behavior)
Ok thank you! Indeed, if I use the Docker storage and provide the
then the flow run is executed correctly! However, in the case that I need to use S3 storage, how can I provide the
? The only way I see is to specify a task definition, which makes sense to me 🙂 Is this right? Thank you!
The flow’s ECSRun allows for setting the
You can either set it there directly or in the task definition if you have a fully custom definition you want to use
Thank you! I read all the arguments but I missread that one!