https://prefect.io
Join Slack
Hey folks, we've recently upgraded from `2.14.5` to `2.16.0` and we have noticed a small issue with ...
# ask-community
m
Hey folks, we've recently upgraded from 
2.14.5
to 
2.16.0
and we have noticed a small issue with our task logs. Within the UI, task logs appear after a decently long delay. After looking into it we found that its the 
api/logs/filter
endpoint that is causing issues. We found that when filtering by 
task_run_id
it would take ~20 seconds, while without filtering it would take ~50ms. I've attached some examples of the bodies here:
Copy code
{"logs":{"level":{"ge_":0},"task_run_id":{"any_":["bb83c087-05bf-4ea5-9404-775c218d2e05"]}},"sort":"TIMESTAMP_ASC","offset":0,"limit":200}


{"logs":{"level":{"ge_":0}},"sort":"TIMESTAMP_ASC","offset":0,"limit":200}
n
hi @Mark Zheng - are you using prefect cloud or hosting server?
m
We are hosting our own server and using postgres
Just an update on this, I think we found the cause to potentially be the number of logs we have, although it is strange that it wasn't a gradual increase in delay but rather a sudden jump. We're going to look into setting an expiry time for log messages stored in the database.
As a side question, does prefect offer any support for this functionality?
n
you might be interested in implementing an additional loop service like this one, one that simply reads / cleans up old runs
did you see a drastic increase in the number of logs you were generating? or just latency
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by