https://prefect.io logo
Title
l

Luke Orland

03/04/2022, 5:48 PM
On Prefect Cloud,
<https://cloud.prefect.io/newsela/flow-run/><flow-run-id>?logs
when I click the Download button the generated file only contains 200 of thousands of log messages. Is that expected?
k

Kevin Kho

03/04/2022, 5:59 PM
Yes it only downloads the currently loaded data. You would need to use the graphql API and pagination to get them all
:thank-you: 1
l

Luke Orland

03/04/2022, 11:31 PM
since I'll need to loop over offset values, I will need to know the max offset. I'm wondering if there's an aggregate graphql field that I can use to request the number of logs for a given flow run, or if I should put the loop in a Python try/except until it receives an API_ERROR?
Where's the best documentation for the graphql API, besides the console / Interactive API?
k

Kevin Kho

03/04/2022, 11:44 PM
So I just tried this and setting the offset high enough just returns an empty list so you can check for that:
query {
  flow_run(where: {id: {_eq: "fce05080-bf90-41fb-857a-4ada7c545b63"}}) {
		logs(limit: 50,offset:50) {
      id
      flow_run_id
      level
      message
      }
		}
  }
does it error for you?
The Interactive API is the only documentation but I find going to the UI repo and seeing how they queried helps a ton so you can look at it here
l

Luke Orland

03/05/2022, 12:02 AM
I also got an empty list.
but the first time I ran this in the Interactive API, I got a timeout error:
then i ran the same again and got
{
  "data": {
    "log": []
  }
}
This is very helpful, Thanks!
k

Kevin Kho

03/05/2022, 12:10 AM
Oh I see. We can try upping the timeout (on the Python client side) if it helps?
l

Luke Orland

03/05/2022, 12:28 AM
I see there's a feature request for the download button to retrieve all the logs: https://github.com/PrefectHQ/ui/issues/487
👍 1
k

Kevin Kho

03/05/2022, 12:37 AM
Sounds good. I guess you’ll get a direct response from the UI team on the issue you opened