jack

    jack

    7 months ago
    What are the pieces required to get ECS logs into cloudwatch? The reason we are interested is because sometimes an ECS task fails to initialize and so there are no prefect logs to inspect.
    Anna Geller

    Anna Geller

    7 months ago
    This should happen automatically if you configure a default log group. How did you start your agent? Do you run your agent as an ECS service? What is your task definition for a flow run?
    jack

    jack

    7 months ago
    Hi @Anna Geller Is it the agent that needs logs enabled?
    Here is the logConfiguration for our agent's task definition
    But the task definitions for the individual tasks that are run by prefect have
    logConfiguration: null
    It turns out that logs from our agent are arriving in cloudwatch 😄 But what we really want is logs from individual tasks
    Anna Geller

    Anna Geller

    7 months ago
    by default Prefect infers those settings from the agent unless you specify something else on the ECSRun. Can you share your ECSRun configuration?
    jack

    jack

    7 months ago
    Here is our call to ECSRun:
    RUN_CONFIG = ECSRun(
        run_task_kwargs={'cluster': 'default'},
        execution_role_arn='arn:aws:iam::00000000000:role/ecsTaskExecutionRole',
        task_role_arn='arn:aws:iam::00000000000000:role/prefect-ecs-task-role',
        labels=['ecs-agent'],
        memory='8 GB',
    )
    Anna Geller

    Anna Geller

    7 months ago
    If this task role
    prefect-ecs-task-role
    has a logging configuration, logging to CloudWatch should work. If not, feel free to build a small reproducible example and submit a Github issue and I can try to replicate and fix when possible. But based on this message (which I misread at first) - this is likely the issue. You need that logConfiguration on your flow run ECS task definition for the logs to appear in CloudWatch
    jack

    jack

    7 months ago
    The task role
    prefect-ecs-task-role
    includes this
    How do we know which log group it will attempt to write to so that we can create the log group ahead of time?
    Anna Geller

    Anna Geller

    7 months ago
    This blog post describes how I approached it. I created a log group manually once and ALL logs are created there, both for the agent and for all flow runs.
    aws logs create-log-group --log-group-name $ECS_LOG_GROUP_NAME --region $AWS_REGION
    so to directly answer your question - you can create a log group and reference the same log group for the agent and for your flow runs and all your logs should eventually land in CloudWatch
    jack

    jack

    7 months ago
    How do we specify a log group in
    ECSRun
    ?
    Kevin Kho

    Kevin Kho

    7 months ago
    Unless Anna corrects me, it needs to be included in the task definition, which then gets registered along with the ECS task
    Anna Geller

    Anna Geller

    7 months ago
    the ideal setup is when you set it on your agent's task definition, then you create ECS service for your agent and then all your flow runs take this config from the agent. You can find more detailed answer and walkthrough in the blog post I shared. I never needed to specify the log group for flow runs manually.
    There are a couple of ECSRun examples here
    Kevin Kho

    Kevin Kho

    7 months ago
    Yeah from experience I haven’t had to fiddle with the ECSRun to get the log group because it just comes in when you define it from the agent. But if you do the Task Definition route, Anna has a good example here of how to get it in the containerDefinition
    k

    Kevin Grismore

    7 months ago
    Hi! I'm working with Jack on this one, so I've been following along. I think we're going to go with custom task definitions so we have logs for different flows sent to different log groups. Thanks for all your help so far!