https://prefect.io logo
Title
d

Dylan Lim

05/24/2022, 2:52 AM
Hello! I’m trying to run a registered flow but I got this error message in the agent logs, could anyone help to explain what this error means?
[2022-05-24 02:45:17,232] ERROR - agent | Failed to query for flow run metadata
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/agent/agent.py", line 328, in _submit_deploy_flow_run_jobs
    flow_runs = self._get_flow_run_metadata(flow_run_ids)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/agent/agent.py", line 688, in _get_flow_run_metadata
    result = self.client.graphql(query)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
k

Kevin Kho

05/24/2022, 2:53 AM
Is this Prefect Cloud or Server?
This is weird. Could you try re-registering?
d

Dylan Lim

05/24/2022, 2:56 AM
I’m using cloud. I tried removing it from the cloud UI and re registering but still got the same error. The flow on the cloud ui just shows that it's pending, not even failing
k

Kevin Kho

05/24/2022, 3:00 AM
Is the Flow simple enough to share?
And does the same thing happen for a simple Flow?
:upvote: 1
d

Dylan Lim

05/24/2022, 3:04 AM
Sure, let me abstract some of the details out:
def build_flow():
    pwd = PrefectSecret("PASSWORD")

    with Flow("etl") as flow:
        data = extract()
        values = transform(data)
        output = load(values, pwd)
    return flow

if __name__ == "__main__":
    flow = build_flow()
    flow.register(project_name="test_runs")
I tested another flow with a similar structure yesterday and it was successful; didn’t encounter this error. I also just tried reauthenticating but ran into the same error message in the agent logs. If it helps - the
extract
task pulls data from an API, the
values
task parses the response and returns a list of tuples, and the
load
task uses
PostgresExecuteMany
to write the list of values into a db
k

Kevin Kho

05/24/2022, 3:08 AM
The only thing that looks off is the secret outside of the Flow. But I think that should error when it goes to the load task, not at the flow start. Could you try maybe: 1. move secret to inside 2. register the flow under a different name to test?
d

Dylan Lim

05/24/2022, 3:11 AM
Just tried both - sadly I’m still getting the same error messages 😞
def build_flow():
    with Flow("etl_renamed") as flow:
        pwd = PrefectSecret("PASSWORD")

        data = extract()
        values = transform(data)
        output = load(values, pwd)
    return flow

if __name__ == "__main__":
    flow = build_flow()
    flow.register(project_name="test_runs")
k

Kevin Kho

05/24/2022, 3:22 AM
That’s very weird, Could you DM me the whole flow?
👍 1
Looks fine. Just a note that:
yesterday = (datetime.date.today() - datetime.timedelta(days=1)).strftime('%Y-%m-%d')
will be fixed once because it’s calculated during registration time. This should be a task to calculate it every time you run the flow. What is the Prefect version of registration and of the agent?
d

Dylan Lim

05/24/2022, 3:35 AM
ah thanks for the spot! Will put that in a task 🙂 For the version of registration, how can I retrieve it? I deleted the previous flow and re-registered so it’s currently just version 1. For the agent, I’m using local
k

Kevin Kho

05/24/2022, 3:40 AM
More like
pip show prefect
d

Dylan Lim

05/24/2022, 3:42 AM
oh right 🤦‍♂️ here it is:
Name: prefect
Version: 1.2.1
Summary: The Prefect Core automation and scheduling engine.
Home-page: <https://www.github.com/PrefectHQ/prefect>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires: click, cloudpickle, croniter, dask, distributed, docker, importlib-resources, marshmallow, marshmallow-oneofschema, msgpack, mypy-extensions, packaging, pendulum, python-box, python-dateutil, python-slugify, pytz, pyyaml, requests, tabulate, toml, urllib3
Required-by:
k

Kevin Kho

05/24/2022, 3:44 AM
I can’t see anything wrong immediately. I’ll leave a message for the team and get back to you tom
d

Dylan Lim

05/24/2022, 3:45 AM
Thanks! Appreciate it @Kevin Kho
a

Anna Geller

05/24/2022, 10:51 AM
You need to move the flow object creation out of main. Currently the flow object is only created when you run flow locally
d

Dylan Lim

05/24/2022, 11:07 AM
Hey @Anna Geller! Thanks for the response - I tried removing the main, creating a new project and re-registering and re-running the agent; but I still goto the same error
a

Anna Geller

05/24/2022, 11:38 AM
try this:
from prefect import task, Flow


with Flow("etl") as flow:
    pwd = PrefectSecret("PASSWORD")
    data = extract()
    values = transform(data)
    output = load(values, pwd)
then from CLI:
prefect register --project test_runs -p yourflow.py
no build_flow function and no creation of flow in main, just define your flow in your script and then register/run from CLI
d

Dylan Lim

05/24/2022, 3:01 PM
Hi @Anna Geller - I’ve tried to do this but the same error message pops up. I also tried refreshing API keys and logging out / relogging in and restarting agents/projects, stripping down to the bare minimum (eg hello world flow), which led to some weird behaviour - two out of 15 or so flows mysteriously triggered but the rest gave the same error messages. Still can’t really find a pattern, so I’ll just sleep on it for now…
Just to update - I have no issues running it on my remote server, but for some reason the
prefect agent local start
doesn’t work on my machine
a

Anna Geller

05/25/2022, 12:31 PM
"X doesn't work" is a bit of an oversimplification giving us very little info to help 😄 can you say more? even better - could you try writing up some README GitHub Gist documenting step by step what you have done so far, what were the results of it and which behavior was surprising to you or didn't work as expected? this would significantly help to see a bigger picture and identify what might be happening
k

Kevin Kho

05/25/2022, 2:42 PM
That’s very weird. But it’s only this flow that fails also right? On a specific machine?
d

Dylan Lim

05/26/2022, 2:30 AM
ah sorry - by doesn’t work I mean that I get the same error messages in this thread (it’s been the same error since the start) And @Kevin Kho strangely it’s all flows; I even opened a new Prefect account and logged in but the same error message shows up. However, when I ssh into my remote server and register flows from there + run the agent there, things work fine 😅 It’s probably something funky on my machine that somehow caused the error
k

Kevin Kho

05/26/2022, 2:35 AM
Do you have another machine you can try too?
d

Dylan Lim

05/26/2022, 2:49 AM
Not at the moment, but I’ll see if I can test it on a friend’s laptop. Just a hunch, but does deleting projects / flows and re-creating them with the same name cause problems?
k

Kevin Kho

05/26/2022, 3:02 AM
Not problems like the one you are facing. Deleting projects is a soft delete so it’s marked for deletion and then actual deletion happens like 15 minutes later maybe. If you try to delete and create immediately, you will get an error saying that the project name already exists. The error you have is bizarre for me. Does it happen for a simple “hello world” flow too?
d

Dylan Lim

05/26/2022, 3:13 AM
@Kevin Kho yep unfortunately 😞 it happens for all flows registered from my machine. To clarify,
flow.run()
works but
flow.register(...)
+
prefect agent local start
doesn’t and produces the
Failed to query for flow run metadata
error
k

Kevin Kho

05/26/2022, 3:14 AM
Can you run
prefect diagnostics
?
d

Dylan Lim

05/26/2022, 3:14 AM
For now, my workaround is to develop and push my code from my laptop but register it and run the agent on our server
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "macOS-12.3.1-x86_64-i386-64bit",
    "prefect_backend": "cloud",
    "prefect_version": "1.2.2",
    "python_version": "3.9.2"
  }
}
k

Kevin Kho

05/26/2022, 3:16 AM
Do you have anything in the config.toml or env var settings?
Spinning up an agent has the same error?
prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
d

Dylan Lim

05/26/2022, 3:19 AM
Yeah - spinning up the agent on my machine seems to always cause this error. I’ll dm you the config.toml contents
k

Kevin Kho

05/26/2022, 2:32 PM
I looked at it and it looked pretty fine. Could not identify anything wrong. You can also try an empty
config.toml
though to be sure you are using all the defaults.
👀 1