https://prefect.io logo
l

liren zhang

12/07/2020, 7:16 PM
Hi Community, We have used graphql to extract prefect logs. The initial approach was just doing getting a list of flow, flow run and task run to iterate through the list. Obviously, it is the least performant way of extracting data from a database. We also started reviewing batching and caching techniques that Facebook uses with their dataloader. in the mean time, I am hoping to check in with the community experts and see if there is a much easier way to extract the logs without even going through graphql interface. Any info would be much appreciated.
z

Zanie

12/07/2020, 7:28 PM
Hi! We’re actually looking into speeding this up in Prefect Cloud soon so we can provide log downloads in the UI. Are you trying to stream the logs or just batch download them for task runs? Hopefully someone else has some advice for you as well!
l

liren zhang

12/07/2020, 8:36 PM
Good to know @Zanie we are looking to do near real time log data extraction. If Prefect can provide an API end point that allow to download the delta log dataset in batches, it will certainly satisfy our needs
We would like to get to the point that we can send an API request and bring back the delta log data set and then push it into a db via bulkload utilities
z

Zanie

12/07/2020, 8:44 PM
What kind of logs are you looking to collect? You may just want to configure a log handler that pushes directly to your own API. You can attach your own handler to the prefect logger
l

liren zhang

01/19/2021, 7:28 PM
Hi experts: I changed my logging to use local time zone in the config.toml. Do I need to restart my Prefect server and/or Agent?
z

Zanie

01/19/2021, 7:43 PM
That’s a bit tricky, I think this will actually be dependent on which logs you’re interested in as new logger instances will use your new timezone but some older instances may not. For consistency across all loggers, I’d recommend restarting them.
l

liren zhang

01/19/2021, 8:30 PM
Hi @Zanie I restarted the server and agent after I change the logging config to the following in my config.toml
Copy code
# additional log attributes to extract from context
# e.g., log_attributes = "['context_var']"
log_attributes = "[]"

#added to use local timezone for the log
display_tz ="US/Eastern"

# the timestamp format
datefmt = "%Y-%m-%d %H:%M:%S%z"

# Send logs to Prefect Cloud
log_to_cloud = false
but seems the log entry I extracted are still showing UTC
not sure if I have done this correctly
z

Zanie

01/19/2021, 9:13 PM
Logs are always stored in the backend as UTC — this timezone is only for display
You’ll have to convert them to your timezone if you are querying for them.
l

liren zhang

01/19/2021, 9:23 PM
ok, thx
2 Views