Zhang David
09/11/2023, 5:12 PMtail -f
the pool process, I can see the logs streamed in but mixed together across all flows so not really readable.
My question is: is there any way that I could stream in the logs separately for each flow ? for me, this is very important for production monitoring and stuff.
Really appreciate if anyone could help!Islam Otmani
09/14/2023, 9:58 PMwatch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'
Zhang David
09/14/2023, 10:09 PMwatch -n 1 'tail -n 15 /path/to/file_or_command | grep "run_name"'In addition, is there any way I can stream the logs from a specific work queue from a work pool? I've actually being searching in the documentation, but couldn't find it.
ps -ef
and get the work pool process PID, and do a tail -f /proc/$PID/fd/1
Islam Otmani
09/14/2023, 10:10 PMZhang David
09/14/2023, 10:11 PMtaking a look ^you mean the documentation above (in addition to the watch cmd) ?
Islam Otmani
09/14/2023, 10:14 PMlogging.yaml
if we can add the work queue name in there, so that you can grep as mentioned aboveZhang David
09/14/2023, 10:17 PMrun_name
would not work - cuz my logs are only showing in console b/c its from my own library/pkg, so not from the prefect logger.Islam Otmani
09/14/2023, 10:32 PMYou can customize logging configuration by creating your own version ofAs I suspected, you're able to use thewith custom settings, by either creating the file at the default location (logging.yml
) or by specifying the path to the file with/.prefect/logging.yml
. (If the file does not exist at the specified location, Prefect ignores the setting and uses the default configuration.)PREFECT_LOGGING_SETTINGS_PATH
logging.yaml
file to customize stuff. You can also configure handlers to write to files etc as needed, this is linked from our docs for further info:
https://docs.python.org/3/library/logging.config.html#logging.config.dictConfigZhang David
09/14/2023, 11:16 PMlogging.yaml
Islam Otmani
09/15/2023, 1:18 AM