Hi there. I have a module that users `loguru` <https://github.com/Delgan/loguru> It's used inside of...

Pedro Machado

12/14/2021, 3:42 PM

Hi there. I have a module that users

loguru

https://github.com/Delgan/loguru It's used inside of a task. When I run the flow locally, I see the loguru output but when it runs remotely (ECS), I don't see the logs in Prefect Cloud. How could I get it to write to the Prefect logs? Also, how should I set up my ECS flow to get it to write the container logs to cloudwatch? Thanks!

Kevin Kho

12/14/2021, 3:47 PM

Hey @Pedro Machado, add the extra loggers here to Prefect for the logs to show up. This will not work for Dask executor.

Kevin Kho

12/14/2021, 3:47 PM

For logs to cloud watch, you need to configure it like this:

Copy code

task_definition = yaml.safe_load(
      """
      cpu: 1024
      memory: 2048
      containerDefinitions:
      - name: flow
        logConfiguration:
          logDriver: awslogs
          options:
            awslogs-group: /ecs/prefect-agent
            awslogs-region: us-east-1
            awslogs-stream-prefix: ecs
            awslogs-create-group: 'true'
      """

Kevin Kho

12/14/2021, 3:51 PM

There is a loguru thread somewhere up here I think in case you are curious

Kevin Kho

12/14/2021, 3:51 PM

Ah I guess it’s gone

Pedro Machado

12/14/2021, 5:17 PM

Thanks Kevin. I think I a missing something. I tried this env var on the ECSRun and got the snowflake connector to log but not my custom library. I am thinking I need to do something on the loguru side. Any ideas?

Copy code

PREFECT__LOGGING__EXTRA_LOGGERS="['snowflake.connector', 'boto3', 'custom_lib']"

Kevin Kho

12/14/2021, 5:19 PM

May try

Copy code

@task(log_stdout=True)

if the logs go to stdout? Or maybe add loguru?

41 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.