Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

What are the pieces required to get ECS logs into cloudwatch?

The reason we are interested is because sometimes an ECS task fails to initialize and so there are no prefect logs to inspect.

image.png

This should happen automatically if you configure a default log group. How did you start your agent? Do you run your agent as an ECS service? What is your task definition for a flow run?

Hi <@U02H1A95XDW>

Is it the agent that needs logs enabled?

Here is the logConfiguration for our agent's task definition

But the task definitions for the individual tasks that are run by prefect have `logConfiguration: null`

It turns out that logs from our agent are arriving in cloudwatch :smile:

But what we really want is logs from individual tasks

by default Prefect infers those settings from the agent unless you specify something else on the ECSRun. Can you share your ECSRun configuration?

Here is our call to ECSRun:
```RUN_CONFIG = ECSRun(
    run_task_kwargs={'cluster': 'default'},
    execution_role_arn='arn:aws:iam::00000000000:role/ecsTaskExecutionRole',
    task_role_arn='arn:aws:iam::00000000000000:role/prefect-ecs-task-role',
    labels=['ecs-agent'],
    memory='8 GB',
)```

If this task role `prefect-ecs-task-role` has a logging configuration, logging to CloudWatch should work. If not, feel free to build a small reproducible example and submit a Github issue and I can try to replicate and fix when possible.

But based on <https://prefect-community.slack.com/archives/C014Z8DPDSR/p1645032397682529?thread_ts=1645031458.240669&amp;cid=C014Z8DPDSR|this message> (which I misread at first) - this is likely the issue. You need that logConfiguration on your flow run ECS task definition for the logs to appear in CloudWatch

The task role `prefect-ecs-task-role` includes this

How do we know which log group it will attempt to write to so that we can create the log group ahead of time?

<https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df|This blog post> describes how I approached it. I created a log group manually once and ALL logs are created there, both for the agent and for all flow runs.
```aws logs create-log-group --log-group-name $ECS_LOG_GROUP_NAME --region $AWS_REGION```

so to directly answer your question - you can create a log group and reference the same log group for the agent and for your flow runs and all your logs should eventually land in CloudWatch

How do we specify a log group in `ECSRun`?

Unless Anna corrects me, it needs to be included in the task definition, which then gets registered along with the ECS task

the ideal setup is when you set it on your agent's task definition, then you create ECS service for your agent and then all your flow runs take this config from the agent. You can find more detailed answer and walkthrough in the blog post I shared. I never needed to specify the log group for flow runs manually.

There are a couple of ECSRun examples <https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows|here>

Yeah from experience I haven’t had to fiddle with the ECSRun to get the log group because it just comes in when you define it from the agent. But if you do the Task Definition route, Anna has a good example <https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/flow_task_definition.yaml|here> of how to get it in the containerDefinition

Hi! I'm working with Jack on this one, so I've been following along. I think we're going to go with custom task definitions so we have logs for different flows sent to different log groups. Thanks for all your help so far!