https://prefect.io logo
Join Slack
Powered by
I'm trying to use the Scrapy framework in conjunct...
# ask-community
k
I'm trying to use the Scrapy framework in conjunction with Prefect 2.0, and Scrapy automatically sets up its own core Python Loggers. When I run a flow in the terminal, I can see the logs coming from Scrapy's loggers like so: 
13:37:49.758 | DEBUG   | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>
However, those logs don't appear in the flow run logs UI. I'm guessing I need to set 
PREFECT_LOGGING_EXTRA_LOGGERS
, but I'm not certain what the right way to do that is. 
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"
seems to have no effect. I also tried 
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"
using the name of the specific logger I wanted to see, but that didn't work either.
✅ 1
a
Can you try adding that to your profile file?
k
is just the name of the python package sufficient? like 
PREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
hm, so if I do 
from prefect.logging import get_logger
and inside the flow: 
get_logger('scrapy')
there is indeed a 
prefect.scrapy
logger, but still nothing gets written to the UI from it
even if I remove 
PREFECT_LOGGING_EXTRA_LOGGERS = "scrapy"
, 
get_logger('scrapy')
still returns a 
prefect.scrapy
logger 🤔
a
can you add it to the profile toml file as I showed?
k
I did
so this looks as expected when I run `prefect config view`: 
PREFECT_LOGGING_EXTRA_LOGGERS='scrapy' (from profile)
then when I run 
scrapy_logger = get_logger('scrapy')
, my 
scrapy_logger.handlers
is 
[]
a
not sure about log handlers, but the logger should be available when you set it as extra logger
generally speaking, the idea with extra loggers is to not use those in your flow - in your flow and task, you should use Prefect logger, but if your scrapy logger natively gives some extra logs at runtime, this will be logged to the Prefect backend as well
279 Views
Open in Slack
PreviousNext
Powered by