https://prefect.io logo
z

Zhang David

09/11/2023, 5:12 PM
Hi guys - I have a question on logging: I have many production flows deployed to one local worker pool - if I
tail -f
the pool process, I can see the logs streamed in but mixed together across all flows so not really readable. My question is: is there any way that I could stream in the logs separately for each flow ? for me, this is very important for production monitoring and stuff. Really appreciate if anyone could help!
i

Islam Otmani

09/14/2023, 9:58 PM
Hi @Zhang David, I hope you're well πŸ™‚! You can change the formatting to your liking as documented here via the file `logging.yml`: https://docs.prefect.io/latest/guides/logs/?h=logging#formatters
As for continuously monitoring a grep text, an alternative command could be:
Copy code
watch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'
z

Zhang David

09/14/2023, 10:09 PM
hey - this is super helpful! let me try that !
watch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'
In addition, is there any way I can stream the logs from a specific work queue from a work pool? I've actually being searching in the documentation, but couldn't find it.
Right now - I'm doing this in a pool way. Basically I would
ps -ef
and get the work pool process PID, and do a
tail -f /proc/$PID/fd/1
*poor way
i

Islam Otmani

09/14/2023, 10:10 PM
I see, are you running the flow locally or are you having to SSH into a box, or exec into kubernetes box?
taking a look ^
z

Zhang David

09/14/2023, 10:11 PM
local box
πŸ‘ 1
taking a look ^
you mean the documentation above (in addition to the watch cmd) ?
i

Islam Otmani

09/14/2023, 10:14 PM
I can't find something about queue names in docs, trying to see if I can find a way, e.g. via
logging.yaml
if we can add the work queue name in there, so that you can grep as mentioned above
z

Zhang David

09/14/2023, 10:17 PM
yeah that'd be ideal. also i actually think the grep by
run_name
would not work - cuz my logs are only showing in console b/c its from my own library/pkg, so not from the prefect logger.
i

Islam Otmani

09/14/2023, 10:32 PM
Gotcha πŸ™‚
You can customize logging configuration by creating your own version of
logging.yml
with custom settings, by either creating the file at the default location (
/.prefect/logging.yml
) or by specifying the path to the file with
PREFECT_LOGGING_SETTINGS_PATH
. (If the file does not exist at the specified location, Prefect ignores the setting and uses the default configuration.)
As I suspected, you're able to use the
logging.yaml
file to customize stuff. You can also configure handlers to write to files etc as needed, this is linked from our docs for further info: https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig
z

Zhang David

09/14/2023, 11:16 PM
mmm... ok thanks! iet me play around more w/ the
logging.yaml
the thing is i don't (yet) know how to handle my own custom logs which are not using prefect logger
i cannot change all my logger to be prefect logger
i

Islam Otmani

09/15/2023, 1:18 AM
Sounds good. Give me a shout if needed!
You can also pass individual prefect environment variables instead of the yaml file, it’s mentioned in the example yaml at the top.