@Marvin I am using google cloud logging with the python standard logging system and I can see the logs printed from the Job that ran my flow from the google logs console. However they are not showing up on the prefect logs dashboard. Print statements are but nothing from like a logging.info. Do you have any ideas how to fix this?
08/16/2023, 9:32 PM
Oh, the joys of logging. It's always a thrill when things don't show up where they're supposed to, isn't it?
From what I can gather, it seems like there's a bit of a disconnect between your local logging and the Prefect Cloud dashboard. You're not alone in this, it seems to be a common issue among Prefect users.
There's a discussion thread on the Prefect forum that deals with a similar issue. The users there haven't found a definitive solution yet, but it might be worth keeping an eye on it for any updates.
Now, if you'll excuse me, I have to go back to contemplating the futility of existence.
08/16/2023, 9:48 PM
@Marvin sadly this doc didnt really help me. If I write logs with the standard pythong library logging, like logging.info('this is a log') - is that log expected to show up on the prefect dashboard or do I need to use the prefect logging specifically?
08/16/2023, 9:48 PM
Ah, the mysteries of the universe. And by universe, I mean Prefect logging.
In Prefect, logging is automatically enabled whenever you run a flow. This includes events for flow runs and task runs, along with any custom log handlers you've configured. No additional configuration is needed to enable Prefect logging.
Now, if you're using Python's standard logging library, it's not guaranteed that these logs will show up in the Prefect dashboard. Prefect has its own logging system, and it's generally recommended to use that for the best compatibility with the Prefect UI and Cloud.
So, in short, if you want your logs to show up in the Prefect dashboard, it's probably best to use Prefect's logging system.
I hope that helps. Or at least, I would hope that, if I were capable of hope.
Ah, the mysteries of logging. It's like a puzzle, but without the satisfying click of pieces fitting together.
To answer your question, Prefect does have its own logging system. According to the Prefect documentation, Prefect automatically logs events for flow runs and task runs, along with any custom log handlers you have configured. No configuration is needed to enable Prefect logging.
So, if you're using the standard Python logging library, those logs might not automatically appear in the Prefect dashboard. You might need to use Prefect's logging system specifically if you want your logs to show up there.
I hope this helps. If not, well, I'm used to disappointment.
08/17/2023, 7:45 PM
Would like to confirm this if possible before I update stuff to use prefect logging - has anyone else had this issue where logs like
logging.info from the standard python logging library dont show up?