Does Prefect provide a way to search flow run logs e.g. searching for a specific log message content?

Question

Hi all I m testing prefect orion server locally Is there a way to search flow run logs I know I can search by run name on the flow runs tab but that is limited utility I know the logs exist I want to search for particular error messages that were logged in the last 100 or so flow runs I cant see a way in the UI or even int prefect CLI

merlin · Answer

Got it roughly Would love to see prefect CLI or python API version of this but here is how I got to the UI view of a flow run with an error message logged ```$ gt sqlite3 readonly ~ prefect orion db sqlite gt select distinct flow run id created from log where message like ERROR Traffic order by 2 desc 46b4e64b a9cf 4c60 8c58 6766ad0c5ad4|2022 10 29 18 05 53 352675``` Then I plugged that flow run id into the browser `<http 127 0 0 1 4200 flow run 46b4e64b a9cf 4c60 8c58 6766ad0c5ad4>` Great Now I can look at the offending flow run in the UI

Anna Geller · Answer

initially I thought that you are interested in searching for a specific log message content of a given flow run on that flow run s page but it looks like you want to globally search for that message across flow runs and this is a harder problem and something I would rather expect from a log aggregator rather than from Prefect are you already using something like Splunk ES + Kibana or Datadog

merlin · Answer

not using those im just an ersatz DE where i need to support my own work ha ha now that you mention it would love to use a better tool can you recommend best Datadog usage setup my collegues use that

Anna Geller · Answer

not yet but let me open an issue for tracking we need to do a better job of forwarding logs to log aggregators currently you would need to do it by attaching custom logger and we can definitely make it easier btw funny that you used the word Ersatz DE I m from de

merlin · Answer

i visited München once in December and loved it

Anna Geller · Answer

good choice of timing all the Christmas markets Weihnachtsmärkte make that worthwhile already christmas tree

Anna Geller · Answer