Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi guys - I have a question on logging:

I have many production flows deployed to one local worker pool - if I `tail -f` the pool process, I can see the logs streamed in but mixed together across all flows so not really readable.
My question is: is there any way that I could stream in the logs separately for each flow ? for me, this is very important for production monitoring and stuff.

Really appreciate if anyone could help!

Hi <@U050V8KUK6Z>,
I hope you're well :slightly_smiling_face:!
You can change the formatting to your liking as documented here via the file `logging.yml`:
<https://docs.prefect.io/latest/guides/logs/?h=logging#formatters>

As for continuously monitoring a grep text, an alternative command could be:
```watch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'```

hey - this is super helpful! let me try that !
&gt;  watch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'
In addition, is there any way I can stream the logs from a specific *work queue* from a work pool?
I've actually being searching in the documentation, but couldn't find it.

Right now - I'm doing this in a pool way.

Basically I would `ps -ef` and get the work pool process PID, and do a `tail -f /proc/$PID/fd/1`

I see, are you running the flow locally or are you having to SSH into a box, or  exec into kubernetes box?

&gt;  taking a look ^
you mean the documentation above (in addition to the watch cmd) ?

I can't find something about queue names in docs, trying to see if I can find a way, e.g. via `logging.yaml` if we can add the work queue name in there, so that you can grep as mentioned above

yeah that'd be ideal.

also i actually think the grep by `run_name`  would not work - cuz my logs are *only* showing in console b/c its from my own library/pkg, so not from the prefect logger.

&gt; You can customize logging configuration by creating your own version of `logging.yml` with custom settings, by either creating the file at the default location (`/.prefect/logging.yml`) or by specifying the path to the file with `PREFECT_LOGGING_SETTINGS_PATH`. (If the file does not exist at the specified location, Prefect ignores the setting and uses the default configuration.)
As I suspected, you're able to use the `logging.yaml` file to customize stuff. You can also configure handlers to write to files etc as needed, this is linked from our docs for further info:
<https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig>

mmm... ok thanks! iet me play around more w/ the `logging.yaml`

the thing is i don't (yet) know how to handle my own custom logs which are not using prefect logger

i cannot change all my logger to be prefect logger

You can also pass individual prefect environment variables instead of the yaml file, it’s mentioned in the example yaml at the top. 