Hi Community, We have used graphql to extract prefect logs. The initial approach was just doing getting a list of flow, flow run and task run to iterate through the list. Obviously, it is the least performant way of extracting data from a database. We also started reviewing batching and caching techniques that Facebook uses with their dataloader. in the mean time, I am hoping to check in with the community experts and see if there is a much easier way to extract the logs without even going through graphql interface. Any info would be much appreciated.

Hi! We’re actually looking into speeding this up in Prefect Cloud soon so we can provide log downloads in the UI. Are you trying to stream the logs or just batch download them for task runs? Hopefully someone else has some advice for you as well!

Good to know @Zanie we are looking to do near real time log data extraction. If Prefect can provide an API end point that allow to download the delta log dataset in batches, it will certainly satisfy our needs

What kind of logs are you looking to collect? You may just want to configure a log handler that pushes directly to your own API. You can attach your own handler to the prefect logger

Hi experts: I changed my logging to use local time zone in the config.toml. Do I need to restart my Prefect server and/or Agent?

That’s a bit tricky, I think this will actually be dependent on which logs you’re interested in as new logger instances will use your new timezone but some older instances may not. For consistency across all loggers, I’d recommend restarting them.

Hi @Zanie I restarted the server and agent after I change the logging config to the following in my config.toml

# additional log attributes to extract from context
# e.g., log_attributes = "['context_var']"
log_attributes = "[]"

#added to use local timezone for the log
display_tz ="US/Eastern"

# the timestamp format
datefmt = "%Y-%m-%d %H:%M:%S%z"

# Send logs to Prefect Cloud
log_to_cloud = false

but seems the log entry I extracted are still showing UTC

Logs are always stored in the backend as UTC — this timezone is only for display

ok, thx