and we have noticed a small issue with our task logs. Within the UI, task logs appear after a decently long delay. After looking into it we found that its the
api/logs/filter
endpoint that is causing issues. We found that when filtering by
task_run_id
it would take ~20 seconds, while without filtering it would take ~50ms. I've attached some examples of the bodies here:
hi @Mark Zheng - are you using prefect cloud or hosting server?
m
Mark Zheng
03/04/2024, 3:51 AM
We are hosting our own server and using postgres
Mark Zheng
03/04/2024, 5:28 AM
Just an update on this, I think we found the cause to potentially be the number of logs we have, although it is strange that it wasn't a gradual increase in delay but rather a sudden jump. We're going to look into setting an expiry time for log messages stored in the database.
Mark Zheng
03/04/2024, 5:51 AM
As a side question, does prefect offer any support for this functionality?
n
Nate
03/04/2024, 2:21 PM
you might be interested in implementing an additional loop service like this one, one that simply reads / cleans up old runs
Nate
03/04/2024, 2:22 PM
did you see a drastic increase in the number of logs you were generating? or just latency
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.