https://prefect.io logo
Does Prefect provide a way to search flow run logs e.g. searching for a specific log message content?
m

merlin

10/29/2022, 5:14 PM
Hi all! I'm testing prefect orion server locally. Is there a way to search flow run logs? I know I can "search by run name" on the flow runs tab, but that is limited utility. I know the logs exist, I want to search for particular error messages that were logged in the last 100 or so flow runs. I cant see a way in the UI, or even int prefect CLI.
2
Got it, roughly. Would love to see prefect CLI or python API version of this but.. here is how I got to the UI view of a flow run with an error message logged:
$> sqlite3 -readonly ~/.prefect/orion.db

sqlite> select distinct flow_run_id, created from log where message like '%ERROR: Traffic%' order by 2 desc;
46b4e64b-a9cf-4c60-8c58-6766ad0c5ad4|2022-10-29 18:05:53.352675
Then I plugged that flow_run_id into the browser:
<http://127.0.0.1:4200/flow-run/46b4e64b-a9cf-4c60-8c58-6766ad0c5ad4>
Great! Now I can look at the offending flow run in the UI!
:gratitude-thank-you: 1
a

Anna Geller

10/29/2022, 6:26 PM
initially, I thought that you are interested in searching for a specific log message content of a given flow run on that flow run's page, but it looks like you want to globally search for that message across flow runs and this is a harder problem and something I would rather expect from a log aggregator rather than from Prefect are you already using something like Splunk, ES + Kibana or Datadog?
m

merlin

10/29/2022, 6:33 PM
not using those, im just an ersatz DE where i need to support my own work ha ha. now that you mention it, would love to use a better tool. can you recommend best Datadog usage setup? my collegues use that
👍 1
a

Anna Geller

10/29/2022, 6:40 PM
not yet, but let me open an issue for tracking, we need to do a better job of forwarding logs to log aggregators -- currently, you would need to do it by attaching custom logger and we can definitely make it easier btw funny that you used the word Ersatz DE - I'm from 🇩🇪
🇩🇪 1
m

merlin

10/29/2022, 6:45 PM
i visited München once in December and loved it.
🙌 1
a

Anna Geller

10/29/2022, 6:50 PM
good choice of timing - all the Christmas markets (Weihnachtsmärkte) make that worthwhile already 🎄
🎯 1