https://prefect.io
Join Slack
Hi all! I've just started using Prefect and I was wondering if it is possible to programmatically ac...
# ask-community
h
Hi all! I've just started using Prefect and I was wondering if it is possible to programmatically access task logs using Python - at the moment I'm using the web interface but it would be great if I could grab and parse them via an API. Is this currently possible? Thanks in advance!
k
Hi @Hayden Sansum, I think something like this will get you started with using the GraphQL to retrieve them:
Copy code
query {
  flow_run(
    where: {flow: {name: {_eq: "gh storage"}}}
    order_by: {created: desc}
  ) {
    name
    id
		flow {
      name
    }
    logs {
      id
      message
      timestamp
    }
  }
}
h
thanks, I'll give that a try
Hey @Kevin Kho - To follow up on my original question - I now have a query which works in the Interactive API and correctly returns all the logs but when I call it using the python client it fails. This is the query itself:
Copy code
query {
  flow(where: { name: { _eq: "flow_name" },
                id: { _eq: "flow_id"}}) {
    id
    name
    flow_runs(where: { state: {_eq: "Success"}, start_time: {_gt: "2021-06-22T00:00:00.000000+00:00"}}
              order_by: { start_time: desc }
      ) {
      id
      state
      start_time
      name
      logs {
        id
        message
        timestamp
      }
    }
  }
}
And when running the same query through python using the prefect Client (
client.graphql
) I get the error:
Copy code
ClientError: [{'path': ['flow'], 'message': 'field "logs" not found in type: \'flow_run\'', 'extensions': {'path': '$.selectionSet.flow.selectionSet.flow_runs.selectionSet.logs', 'code': 'validation-failed', 'exception': {'message': 'field "logs" not found in type: \'flow_run\''}}}]
Do you have any ideas on why this might be occurring?
k
Hey @Hayden Sansum, looking at this
This works for me when I plug in a flow name and id. What version of Prefect are you on? Just so I can try.
h
Thanks for looking into this - i'm using 
v0.14.2
but I've also tried on 
v0.14.22
and I get the same results either way
k
Ok will ask the team
h
For context the python client query method works if I remove the:
Copy code
logs {
        id
        message
        timestamp
      }
from the query so I know it is at least correctly querying the flow but when I try to also grab the logs I get the error
k
Are you on Server or Cloud?
h
I believe Cloud
k
As in you go to cloud.prefect.io to use Prefect right?
h
yes that is correct - that's where I'm accessing the Interactive API from to test the query
k
I can’t reproduce this. Could you try again and confirm this is a persisting issue?
h
It's definitely persistent - I encountered it this morning and I've been trying to find a workaround to no avail so far. If I remove the logs part of the query in python it returns all the correct flow run names, and when I run the full query through the interactive API it works correctly. It's only the combination of running the full query in python including requesting logs that causes the error message above.
k
We’ll look into this but as a workaround, could you query the logs like this?
Copy code
query {
  log(where:{flow_run_id: {_eq: "<<>>"}}) {
    message
    timestamp
    id
    level
  }
}
h
I get the error:
Copy code
ClientError: [{'path': ['log'], 'message': 'field "log" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.log', 'code': 'validation-failed', 'exception': {'message': 'field "log" not found in type: \'query_root\''}}}]
k
Ok we’ll look into it
h
thank you - also this is another case where the query works through the interactive API but not through the Python client
k
Can you try making a new API token? Wondering if it’s authentication related
h
I can check in on that and get back to you - might be tomorrow before I'll be able to test
k
Sure sounds good. This might me because you have an old token that doesnt have permissions for the logs table
h
ah I see, okay I'll investigate and let you know tomorrow if that seems to be causing the issue - thanks again for looking into this
25 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by