m

    Matt Liszewski

    1 year ago
    Hi! We are shipping our Kubernetes Run logs to Datadog. Datadog (as well as most log aggregators) prefers logs in JSON format. What is the best way to get Prefect logs formatted in JSON?
    Dylan

    Dylan

    1 year ago
    Hi @Matt Liszewski! You’re in luck, they already are! Try fetching the logs for a particular Flow Run from our GraphQL API using the interactive API and a query like this:
    query {
      flow_run_by_pk(id: "FLOW_RUN_ID_HERE") {
        id
        name
        state
        logs {
          id
          timestamp
          level
          message
        }
      }
    }
    If you’re shipping logs while the flow is running, take a look at: https://docs.prefect.io/core/idioms/logging.html#configure-the-logger
    I believe these are datadog’s logs re: their python logger: https://docs.datadoghq.com/logs/log_collection/python/?tab=jsonlogformatter
    m

    Matt Liszewski

    1 year ago
    thanks. the issue is Datadog prefers the logs in a JSON format but there is no straightforward way to change the formatter on the Prefect logs. I was able to get it to work on a "local" run by overwriting the Stream handler but that doesn't work when running in a container. I also can't seem to find a working example of attaching a custom log handler. Are there any examples out there of people successfully getting Prefect flows executing in a container to output JSON logs
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Matt Liszewski -- just want to confirm that you need the Prefect logs rather than just your own personal logs? Also, is there a reason you can't pull the logs from the database, format them, and send them to datadog?
    m

    Matt Liszewski

    1 year ago
    I basically want the std.out logs that appear on our containers running the jobs formatted in JSON. We have Datadog setup to pull these logs from our Kubernetes cluster where the jobs are executing. The logs show up but it requires custom configuration for us to parse them in Datadog and prevents us from easily adding custom
    extra
    log fields that we may want to trend or alert on in Datadog
    I could pull them from the api but it seems like it should be possible to format logs as json since most log aggregation platforms (Datadog, Spunk, etc) prefer JSON formatted logs
    Michael Adkins

    Michael Adkins

    1 year ago
    I understand now 🙂 Dylan and I will check in with some more of the devs and see if we can find an ergonomic way to do this.
    m

    Matt Liszewski

    1 year ago
    Awesome thanks
    Adam

    Adam

    1 year ago
    @Matt Liszewski did you manage to get this to work? We’re trying to do the same. We don’t see any logs from stdout from Kubernetes Jobs