Robert Banick

09/12/2023, 12:28 PM
Hi all, We are running Prefect 2.10.x to orchestrate ETL jobs over an AWS stack. We run large, long-running ETLs of climate data that involve lots of third party libraries to interface with various specialized APIs and the like. We are running into an issue where Prefect forwards overly verbose logs to Cloudwatch, costing us $$$ every day. While our ETL script only sends INFO logs, the APIs it interfaces and many libraries it uses like
are spamming our Cloudwatch logs with DEBUG statements. Thi happens in sufficient volumes to make the logs much harder to use and cost us real money on a monthly basis. My hunch is that Prefect is somehow catching these logs and forwarding them on to Cloudwatch. I’ve tried setting
on the Dockerfile of the Agent we use to orchestrate but this had no effect. Should I be setting the root level on the container Dockerfile? Or is there something elsewhere I’m missing? Any help would be most appreciated. Note that I’m concerned about Cloudwatch, not about Prefect UI — I get only the INFO level ETL logs on the Prefect UI, as intended. Cheers, Robert