https://prefect.io logo
d

Damian

08/06/2023, 7:24 PM
Hi, I'm trying to get some flow logs from my local deployment into prefect cloud. In my flow I'm calling
scrapy
to do some web scraping. I see my
scrapy
logs in the local prefect agent logs but I can't manage to have any of them show up in prefect cloud unless I explicitly use prefects loggger with
get_run_logger
. I've tried increasing all log levels (e.g.
PREFECT_LOGGING_LEVEL='INFO'
,
PREFECT_LOGGING_SERVER_LEVEL='INFO'
) and I've also explicitly added
PREFECT_LOGGING_EXTRA_LOGGERS=scrapy
but nothing appears to work. The prefect agent obviously receives the scrapy logs but it won't add them to the specific run logger. What would be the best way to solve this? Another point - there appears to be an error in the docs:
PREFECT_LOGGING_HANDLERS_API_LEVEL
doesn't appear to exist (anymore?) but it's mentioned here. This probably requires an update.
j

Jeff Hale

08/07/2023, 4:16 PM
Does setting
log_prints=True
in the @flow decorator work?
d

Damian

08/09/2023, 6:29 AM
Adding
log_prints=True
to the decorator doesn't do anything in my case. As per the docs this only has an effect for the python builtin
print
.
I have figured out the issue in the meantime - If my
scrapy
code runs in a
flow
, then specifying
PREFECT_LOGGING_EXTRA_LOGGERS
has no effect, no logs appear in Prefect Cloud. If my
scrapy
code runs in a
task
, then setting
PREFECT_LOGGING_EXTRA_LOGGERS=scrapy
forwards the logs nicely to Prefect Cloud. I'm not sure if this is a feature or a bug but this behaviour is certainly not in the docs.
🙌 1