https://prefect.io logo
#prefect-community
Title
# prefect-community
h

Haseeb Ahmad

04/06/2022, 3:01 PM
Hi guys. We're trying to set our log format to log everything JSON so we can consume it in Datadog and search the logs. We'd targetting the following format:
Copy code
{
  "flow_name": "",
  "flow_id": "",
  "task_name": "",
  "task_id": "",
  "message": "",
  "customer_id": ""
}
The questions we have are: 1. Is using
PREFECT__LOGGING__FORMAT
the correct place to specify the JSON format? 2. How can we handle this additional attribute
customer_id
. Our intent here is for most tasks that are related to our customers to do something like
<http://logger.info|logger.info>('Something', customer_id)
3. To the previous point, how can we extend this further such that we can add arbitrary parameters or objects to the log,
<http://logger.info|logger.info>('Something', my_object,my_second_object)
and have them show up nsted in the json?
k

Kevin Kho

04/06/2022, 3:11 PM
I believe you can’t change the logging format for Cloud runs. I think the closest thing you can do is outputting these with the standard Prefect logger. If you have a function that formats the message, you can use it in the task.
Copy code
def write_log(message):
    res = {"flow_name": prefect.context.flow_name,
     "flow_id": prefect.context.flow_id
     ..
     "message": message
    }
    return res

@task
def some_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(write_log(message))
For customer_id, is it’s a parameter, you can get it with
prefect.context.parameters["customer_id"]
. If it’s not a parameter to your flow, then you need to write it when doing
<http://logger.info|logger.info>()
For the arbitrary parameters, you could just accept **kwargs in the write_log function above and format it yourself
12 Views