I m trying to use the Scrapy framework in conjunction with P

Question

I m trying to use the Scrapy framework in conjunction with Prefect 2 0 and Scrapy automatically sets up its own core Python Loggers When I run a flow in the terminal I can see the logs coming from Scrapy s loggers like so `13 37 49 758 | DEBUG | scrapy core scraper Scraped from lt 200 <https quotes toscrape com page 9 > gt ` However those logs don t appear in the flow run logs UI I m guessing I need to set `PREFECT LOGGING EXTRA LOGGERS` but I m not certain what the right way to do that is `prefect config set PREFECT LOGGING EXTRA LOGGERS= scrapy ` seems to have no effect I also tried `prefect config set PREFECT LOGGING EXTRA LOGGERS= scrapy core engine ` using the name of the specific logger I wanted to see but that didn t work either

Anna Geller · Answer

Can you try adding that to your profile file

Kevin Grismore · Answer

is just the name of the python package sufficient like `PREFECT LOGGING EXTRA LOGGERS = scrapy `

Kevin Grismore · Answer

hm so if I do `from prefect logging import get logger` and inside the flow `get logger scrapy ` there is indeed a `prefect scrapy` logger but still nothing gets written to the UI from it

Kevin Grismore · Answer

even if I remove `PREFECT LOGGING EXTRA LOGGERS = scrapy ` `get logger scrapy ` still returns a `prefect scrapy` logger thinking face

Anna Geller · Answer

can you add it to the profile toml file as I showed

Kevin Grismore · Answer

I did

Kevin Grismore · Answer

so this looks as expected when I run `prefect config view` `PREFECT LOGGING EXTRA LOGGERS= scrapy from profile `

Kevin Grismore · Answer

then when I run `scrapy logger = get logger scrapy ` my `scrapy logger handlers` is ` `

Anna Geller · Answer

not sure about log handlers but the logger should be available when you set it as extra logger

Anna Geller · Answer

generally speaking the idea with extra loggers is to not use those in your flow in your flow and task you should use Prefect logger but if your scrapy logger natively gives some extra logs at runtime this will be logged to the Prefect backend as well