Hey everyone! In our setup our stdout logs are getting shipped to DataDog and parsed to determine if they are errors or warnings, so I need to change the format of the log messages to match the parser in DataDog. I know how to do this with the logging library but I'm not sure what the smart way of doing this in a decorator task in production (no Cloud for now just cron for testing): 1. Is it possible to change the timestamp format? 2. Is it possible to change the level within a single flow?

Hey @Jacob Blanco! Great questions; both of these are possible. Before I answer your immediate questions, note that Prefect uses a

toml

configuration file for many things such as this (more here: https://docs.prefect.io/core/concepts/configuration.html) 1. To change the timestamp format you can overwrite this variable in Prefect’s configuration: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L82 Note that you can do this through a user file or an environment variable

PREFECT__LOGGING__FORMAT

2. Similarly, you can update the Logging Level via a user configuration file or environment variable

PREFECT__LOGGING__LEVEL

. Because the loggers are configured at import time, you’ll need to set these variables prior to running the Flow in a python process

Hey @Chris White Thank you for the response. I guess this variable changes the default for all Prefect flows in the same environment. I'm not clear on how to use that to change the timestamp format with that setting. Maybe I'm thinking too much in terms of the {logging} library, but there it's set in a separate variable

datefmt

. Am I missing something? The log config for {logging} looks something like this:

"formatters": {
  "datadog": {
     "class": "logging.Formatter",
    "datefmt": "%Y-%m-%d %H:%M:%S.000000%z",
    "format": "%(asctime)s %(levelname)s %(name)s: %(message)s"
  }
},

Are

datefmt

and

format

"merged" in the Prefect config?

Oh I’m sorry you’re 100% correct. We should expose

datefmt

as a configuration option as well; I can get that in for our next release (which will go out next week). In the meantime you will need to manually configure this on the prefect logger

Amazing, thank you!

👍 anytime! I’ll ping you here when the PR is up

One more question, when you say configure it manually do you mean pull the logger from the context and configure the handler/formatter? Would this also update the flow-level logging format?

yes exactly; if you aren’t using an Agent to run your flows (and are instead calling

flow.run()

) you could run this configuration step just before

flow.run