Hi all, I've lost the ability to retrieve logs us...
# ask-community
j
Hi all, I've lost the ability to retrieve logs using the CLI tool:
Copy code
agent@prefect-agent:~$ prefect get logs --name granite-orca > log.log
Traceback (most recent call last):
  File "/home/agent/.pyenv/versions/3.7.3/bin/prefect", line 10, in <module>
    sys.exit(cli())
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/cli/get.py", line 430, in logs
    result = Client().graphql(query)
  File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
I'm not sure what's going on, I had no trouble until recently. The Web UI can still retrieve logs, but it takes too long to scroll through them as it only keeps 100 entries in memory at a time.
k
Hey @Jeremy Phelps, could you share the query you’re trying to run?
j
Whatever query
prefect get logs --name granite-orca
generates.
k
Ok will take a look
So there’s just too many logs for the query to retrieve with the API, the feedback is to use the Client and limit the logs being retrieved with something like
Copy code
logs_query = {
            with_args(
                "logs",
                {
                    "order_by": {EnumValue("timestamp"): EnumValue("asc")},
                    "where": {
                        "_and": [
                            {"timestamp": {"_lte": END_TIME}},
                            {"timestamp": {"_gt": START_TIME}},
                        ]
                    },
                },
            ): {"timestamp": True, "message": True, "level": True}
        }
        result = client.graphql(
            {
                "query": {
                    with_args(
                        "flow_run",
                        {
                            "where": {"id": {"_eq": FLOW_RUN_ID}},
                        },
                    ): logs_query
                }
            }
        )
        # Unpack the result
        logs = result.get("data", {}).get("flow_run", [{}])[0].get("logs", [])
j
Why can't the CLI tool do something smarter, like using that same query to fetch the logs in chunks?
I notice the query has "limit" and "offset" parameters, which are used by the Web UI.
k
Will ask the team about adding those options to the CLI and opening a ticket for it
@Marvin open “Add offset and limit to
prefect get logs
CLI command”