Hi all, I'm looking at configuring logging for our prefect setup. We use [structlog](https://www.structlog.org) across our python services to format our log entries which are typically sent to cloudwatch log groups; these are then ingested by centralised logging (ELK stack). I'd like to know if it's possible to intercept all logs getting sent to Prefect cloud and duplicate them to stdout, applying formatting (we use JSON). Probably a complicated request; any help much appreciated!

Hey @Will, I took a look, so intercepting it is hard. I think the approach is to add the structlog logger onto the Prefect logger so that the logs reach both places. Per the FAQ ,you can create a handler like this:

import logging

class StructlogHandler(logging.Handler):
    """
    Feeds all events back into structlog.
    """
    def __init__(self, *args, **kw):
        super(StructlogHandler, self).__init__(*args, **kw)
        self._log = structlog.get_logger()

    def emit(self, record):
        self._log.log(record.levelno, record.msg, name=record.name)

and then you can try adding it to the Prefect logger like this . This is the best example I have I think. It adds a FileHandler. In your case, you would add the StructlogHandler

Hey @Kevin Kho, thanks for the prompt response. We have an internal library that functions similarly to this, works well for our services. However, the problem will be collecting the agent logs and logs not generated directly by our code (eg. "Downloading flow" etc); and also it'll be quite verbose to attach the logger on every flow run. Ideally, there would be an option to inject middleware into the agent; I'm guessing this might be tricky and require me to modify some internals

So on the verbosity, if you store your flow as a script, I think you should only need to attach it once. You are right the Flow logs will be covered here, but the agent won’t. These are separate processes and there’s exposed way to add it to the agent. The best bet is maybe using the Python interface to start an agent and attaching it there?

So we can already do this, collecting the logs and sending to the centralised logging is no problem, as I have a lambda function that can handle unstructured logs and convert them to JSON structured events. The problem here though is that we will not be able to add any other context variables to the log events. I think as you say, using the python interface may have to be the way to go here; I'll have a look at what I can do with it

@Will I have a similar issue. I think the right way to handle this is to wrap the cli and insert your logging handlers before calling the cli to setup the agent. Although, it might be that you have to write your own wrapper around the agent (it's an importable library, and it's documented on the API docs somewhere) and insert your log handlers there. Either way, we have to force our way into the playground before it's running somehow, and I think it's one of either of those two ways.

Great, I'm interested to see how that turns out, does seem like the only way to go