https://prefect.io logo
#prefect-community
Title
# prefect-community
k

Kevin Grismore

07/12/2022, 5:58 PM
I'm trying to use the Scrapy framework in conjunction with Prefect 2.0, and Scrapy automatically sets up its own core Python Loggers. When I run a flow in the terminal, I can see the logs coming from Scrapy's loggers like so:
13:37:49.758 | DEBUG   | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>
However, those logs don't appear in the flow run logs UI. I'm guessing I need to set
PREFECT_LOGGING_EXTRA_LOGGERS
, but I'm not certain what the right way to do that is.
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"
seems to have no effect. I also tried
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"
using the name of the specific logger I wanted to see, but that didn't work either.
1
a

Anna Geller

07/12/2022, 6:04 PM
Can you try adding that to your profile file?
k

Kevin Grismore

07/12/2022, 6:07 PM
is just the name of the python package sufficient? like
PREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
hm, so if I do
from prefect.logging import get_logger
and inside the flow:
get_logger('scrapy')
there is indeed a
prefect.scrapy
logger, but still nothing gets written to the UI from it
even if I remove
PREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
,
get_logger('scrapy')
still returns a
prefect.scrapy
logger 🤔
a

Anna Geller

07/12/2022, 6:43 PM
can you add it to the profile toml file as I showed?
k

Kevin Grismore

07/12/2022, 6:43 PM
I did
so this looks as expected when I run `prefect config view`:
PREFECT_LOGGING_EXTRA_LOGGERS='scrapy' (from profile)
then when I run
scrapy_logger = get_logger('scrapy')
, my
scrapy_logger.handlers
is
[]
a

Anna Geller

07/13/2022, 10:47 AM
not sure about log handlers, but the logger should be available when you set it as extra logger
generally speaking, the idea with extra loggers is to not use those in your flow - in your flow and task, you should use Prefect logger, but if your scrapy logger natively gives some extra logs at runtime, this will be logged to the Prefect backend as well
136 Views