Hi guys I have a question on logging I have many production

Question

Hi guys I have a question on logging I have many production flows deployed to one local worker pool if I `tail f` the pool process I can see the logs streamed in but mixed together across all flows so not really readable My question is is there any way that I could stream in the logs separately for each flow for me this is very important for production monitoring and stuff Really appreciate if anyone could help

Islam Otmani · Answer

Hi < Zhang David> I hope you re well slightly smiling face You can change the formatting to your liking as documented here via the file `logging yml`

Islam Otmani · Answer

As for continuously monitoring a grep text an alternative command could be ```watch n 1 tail n 15 path to file or command | grep run name ```

Zhang David · Answer

hey this is super helpful let me try that gt watch n 1 tail n 15 path to file or command | grep run name In addition is there any way I can stream the logs from a specific work queue from a work pool I ve actually being searching in the documentation but couldn t find it

Zhang David · Answer

Right now I m doing this in a pool way Basically I would `ps ef` and get the work pool process PID and do a `tail f proc $PID fd 1`

Zhang David · Answer

poor way

Islam Otmani · Answer

I see are you running the flow locally or are you having to SSH into a box or exec into kubernetes box

Islam Otmani · Answer

taking a look ^

Zhang David · Answer

local box

Zhang David · Answer

gt taking a look ^ you mean the documentation above in addition to the watch cmd

Islam Otmani · Answer

I can t find something about queue names in docs trying to see if I can find a way e g via `logging yaml` if we can add the work queue name in there so that you can grep as mentioned above

Zhang David · Answer

yeah that d be ideal also i actually think the grep by `run name` would not work cuz my logs are only showing in console b c its from my own library pkg so not from the prefect logger

Islam Otmani · Answer

Gotcha slightly smiling face

Islam Otmani · Answer

gt You can customize logging configuration by creating your own version of `logging yml` with custom settings by either creating the file at the default location ` prefect logging yml` or by specifying the path to the file with `PREFECT LOGGING SETTINGS PATH` If the file does not exist at the specified location Prefect ignores the setting and uses the default configuration As I suspected you re able to use the `logging yaml` file to customize stuff You can also configure handlers to write to files etc as needed this is linked from our docs for further info <https docs python org 3 library logging config html logging config dictConfig>

Zhang David · Answer

mmm ok thanks iet me play around more w the `logging yaml`

Zhang David · Answer

the thing is i don t yet know how to handle my own custom logs which are not using prefect logger

Zhang David · Answer

i cannot change all my logger to be prefect logger

Islam Otmani · Answer

Sounds good Give me a shout if needed

Islam Otmani · Answer

You can also pass individual prefect environment variables instead of the yaml file it s mentioned in the example yaml at the top