Hi everyone, are results returned from the API sorted by
created
(or any other field) guaranteed to be returned in the same order? Thanks!
k
Kevin Kho
01/12/2022, 10:24 PM
Could you show me a query you are trying to do? I think you can supply an order by
d
Daniel Komisar
01/12/2022, 10:29 PM
Sorry I don’t think I explained it well. When you sort by different attributes, I believe the same query executed at different times can result in different results being returned due to timing. This is expected, for example if you sort on the timestamp of the logs, those timestamps were created on the agent machine and are not guaranteed to be received by the server in the same order. So if you query for logs that are very new and sort them by timestamp you may see some results out of order. My question are there any attributes that the server guarantees this doesn’t happen for? I thought maybe it was
created
since that seems like a server timestamp. Or is there some counter field I missed? Thank you.
k
Kevin Kho
01/12/2022, 10:34 PM
Ah I see what you mean. Yes for logs specifically, there is a batching process to write them on Cloud. But when it reaches Cloud, it should be ordered correctly when ordering by
timestamp
d
Daniel Komisar
01/12/2022, 11:16 PM
Makes sense, so you guarantee the batches can’t be reordered? Is the same true for other fields also? We’re starting to think about monitoring/analysis but we don’t need anything timely so will probably be doing our own batching and I want to make sure I understand the ordering behavior. Thanks again.
k
Kevin Kho
01/13/2022, 12:20 AM
I’m pretty sure the timestamp is the log timestamp that your python process emits to Cloud to that can’t be fudged. It goes down to subsecond I think.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.