What are the pieces required to get ECS logs into cloudwatch? The reason we are interested is because sometimes an ECS task fails to initialize and so there are no prefect logs to inspect.

This should happen automatically if you configure a default log group. How did you start your agent? Do you run your agent as an ECS service? What is your task definition for a flow run?

Hi @Anna Geller Is it the agent that needs logs enabled?

by default Prefect infers those settings from the agent unless you specify something else on the ECSRun. Can you share your ECSRun configuration?

Here is our call to ECSRun:

RUN_CONFIG = ECSRun(
    run_task_kwargs={'cluster': 'default'},
    execution_role_arn='arn:aws:iam::00000000000:role/ecsTaskExecutionRole',
    task_role_arn='arn:aws:iam::00000000000000:role/prefect-ecs-task-role',
    labels=['ecs-agent'],
    memory='8 GB',
)

If this task role

prefect-ecs-task-role

has a logging configuration, logging to CloudWatch should work. If not, feel free to build a small reproducible example and submit a Github issue and I can try to replicate and fix when possible. But based on this message (which I misread at first) - this is likely the issue. You need that logConfiguration on your flow run ECS task definition for the logs to appear in CloudWatch

The task role

prefect-ecs-task-role

includes this

This blog post describes how I approached it. I created a log group manually once and ALL logs are created there, both for the agent and for all flow runs.

aws logs create-log-group --log-group-name $ECS_LOG_GROUP_NAME --region $AWS_REGION

How do we specify a log group in

ECSRun

?

Unless Anna corrects me, it needs to be included in the task definition, which then gets registered along with the ECS task

the ideal setup is when you set it on your agent's task definition, then you create ECS service for your agent and then all your flow runs take this config from the agent. You can find more detailed answer and walkthrough in the blog post I shared. I never needed to specify the log group for flow runs manually.

Yeah from experience I haven’t had to fiddle with the ECSRun to get the log group because it just comes in when you define it from the agent. But if you do the Task Definition route, Anna has a good example here of how to get it in the containerDefinition

Hi! I'm working with Jack on this one, so I've been following along. I think we're going to go with custom task definitions so we have logs for different flows sent to different log groups. Thanks for all your help so far!