Damian
08/06/2023, 7:24 PMscrapy to do some web scraping. I see my scrapy logs in the local prefect agent logs but I can't manage to have any of them show up in prefect cloud unless I explicitly use prefects loggger with get_run_logger. I've tried increasing all log levels (e.g. PREFECT_LOGGING_LEVEL='INFO', PREFECT_LOGGING_SERVER_LEVEL='INFO' ) and I've also explicitly added PREFECT_LOGGING_EXTRA_LOGGERS=scrapy but nothing appears to work. The prefect agent obviously receives the scrapy logs but it won't add them to the specific run logger. What would be the best way to solve this?
Another point - there appears to be an error in the docs: PREFECT_LOGGING_HANDLERS_API_LEVEL doesn't appear to exist (anymore?) but it's mentioned here. This probably requires an update.Jeff Hale
08/07/2023, 4:16 PMlog_prints=True in the @flow decorator work?Damian
08/09/2023, 6:29 AMlog_prints=True to the decorator doesn't do anything in my case. As per the docs this only has an effect for the python builtin print.Damian
08/09/2023, 6:38 AMscrapy code runs in a flow, then specifying PREFECT_LOGGING_EXTRA_LOGGERS has no effect, no logs appear in Prefect Cloud. If my scrapy code runs in a task , then setting PREFECT_LOGGING_EXTRA_LOGGERS=scrapy forwards the logs nicely to Prefect Cloud. I'm not sure if this is a feature or a bug but this behaviour is certainly not in the docs.