Thread
#prefect-community
    e

    Eddie Atkinson

    1 year ago
    Hi gang, I am playing around with getting data logged to the stdout to be displayed in both the prefect logs and my team’s cloudwatch logs. Does anyone have any ideas how to do this? The flow is super simple but the logs aren’t showing up in cloudwatch:
    from prefect.run_configs import ECSRun
    from prefect import task, Flow
    from prefect.storage import S3
    import prefect
    
    TASK_ARN = "an arn"
    STORAGE = S3(bucket="my-cool-bucket")
    LABELS = ["a label"]
    
    RUN_CONFIG = ECSRun(
        labels=LABELS,
        task_role_arn=TASK_ARN,
        image="prefecthq/prefect:latest-python3.8",
        memory=512,
        cpu=256,
    )
    
    
    @task(log_stdout=True)
    def bye_world():
        print("bye world")
    
    
    @task(log_stdout=True)
    def hello_world():
        print("Hello, World!")
    
    
    with Flow("test_flow", run_config=RUN_CONFIG, storage=STORAGE) as flow:
        hello_world()
        bye_world()
    
    
    flow.register(project_name="Tesla")
    My hunch is that I need to get a log group attached to my task definition. However, it looks like prefect creates a new task definition on each run, so I guess I need to specify the details of that?
    Never mind, I figured it out. For anyone who is interested I created another ECS task, attached a log group, and then specified the task definition family as the
    task_definition_arn
    in the call to
    ECSRun
    . I’m using the serverless framework and here is the config for those that are interested:
    tasks:
          prefect-task:
            name: ${self:custom.ecsTaskName}-prefect-task
            image: 'prefecthq/prefect:latest-python3.8'
            desired: 0
            cpu: 256 # 0.25vcpu
            memory: 512 # 512mb
            override:
              role: ${self:custom.ecsBaseTaskRoleName}
              container: 
                Name: flow 
                # container must be called 'flow' for prefect to run tasks 
                # when using ECSRun: <https://docs.prefect.io/api/latest/run_configs.html#ecsrun>
                LogConfiguration:
                  LogDriver: "awslogs"
                  Options:
                    awslogs-group: ${self:custom.ecsClusterLogGroupName}
                    awslogs-region: ${self:custom.region}
                    awslogs-stream-prefix: ${self:custom.ecsTaskLogPrefix}
    With this flow:
    RUN_CONFIG = ECSRun(
        labels=LABELS,
        task_definition_arn=TASK_DEFINITION_FAMILY,
        memory=512,
        cpu=256,
    )
    
    
    @task(log_stdout=True)
    def bye_world():
        print("bye world")
    
    
    @task(log_stdout=True)
    def hello_world():
        print("Hello, World!")
    
    
    with Flow("a_flow", run_config=RUN_CONFIG, storage=STORAGE) as flow:
        hello_world()
        bye_world()
    Kevin Kho

    Kevin Kho

    1 year ago
    @Marvin archive “Log ECS Task to CloudWatch”
    Marvin

    Marvin

    1 year ago
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Eddie Atkinson, thanks for circling back. I was gonna suggest doing it on the agent side like this , but I think the run config is better.
    e

    Eddie Atkinson

    1 year ago
    Hi @Kevin Kho, thanks for your reply. I think my approach works best for my use case as members of my team can now define arbitrary flows to run with that task definition and always know that they’re going to get logs on Cloudwatch without needing to fiddle with task definitions themselves. P.S. I think it’s great that Prefect is open source, being able to read through source code is sometimes a great complement to the documentation!
    Kevin Kho

    Kevin Kho

    1 year ago
    I agree with the approach 👍 And yes, the codebase is pretty legible so we have a lot of users subclassing things