Kevin Grismore
07/12/2022, 5:58 PM13:37:49.758 | DEBUG | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>
However, those logs don't appear in the flow run logs UI. I'm guessing I need to set PREFECT_LOGGING_EXTRA_LOGGERS
, but I'm not certain what the right way to do that is.
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"
seems to have no effect. I also tried prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"
using the name of the specific logger I wanted to see, but that didn't work either.Anna Geller
07/12/2022, 6:04 PMKevin Grismore
07/12/2022, 6:07 PMPREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
from prefect.logging import get_logger
and inside the flow:
get_logger('scrapy')
there is indeed a prefect.scrapy
logger, but still nothing gets written to the UI from itPREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
,
get_logger('scrapy')
still returns a prefect.scrapy
logger 🤔Anna Geller
07/12/2022, 6:43 PMKevin Grismore
07/12/2022, 6:43 PMPREFECT_LOGGING_EXTRA_LOGGERS='scrapy' (from profile)
scrapy_logger = get_logger('scrapy')
, my scrapy_logger.handlers
is []
Anna Geller
07/13/2022, 10:47 AM