Damian
08/06/2023, 7:24 PMscrapy
to do some web scraping. I see my scrapy
logs in the local prefect agent logs but I can't manage to have any of them show up in prefect cloud unless I explicitly use prefects loggger with get_run_logger
. I've tried increasing all log levels (e.g. PREFECT_LOGGING_LEVEL='INFO'
, PREFECT_LOGGING_SERVER_LEVEL='INFO'
) and I've also explicitly added PREFECT_LOGGING_EXTRA_LOGGERS=scrapy
but nothing appears to work. The prefect agent obviously receives the scrapy logs but it won't add them to the specific run logger. What would be the best way to solve this?
Another point - there appears to be an error in the docs: PREFECT_LOGGING_HANDLERS_API_LEVEL
doesn't appear to exist (anymore?) but it's mentioned here. This probably requires an update.Jeff Hale
08/07/2023, 4:16 PMlog_prints=True
in the @flow decorator work?Damian
08/09/2023, 6:29 AMlog_prints=True
to the decorator doesn't do anything in my case. As per the docs this only has an effect for the python builtin print
.scrapy
code runs in a flow
, then specifying PREFECT_LOGGING_EXTRA_LOGGERS
has no effect, no logs appear in Prefect Cloud. If my scrapy
code runs in a task
, then setting PREFECT_LOGGING_EXTRA_LOGGERS=scrapy
forwards the logs nicely to Prefect Cloud. I'm not sure if this is a feature or a bug but this behaviour is certainly not in the docs.