Dylan Lim
05/24/2022, 2:52 AM[2022-05-24 02:45:17,232] ERROR - agent | Failed to query for flow run metadata
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/agent/agent.py", line 328, in _submit_deploy_flow_run_jobs
flow_runs = self._get_flow_run_metadata(flow_run_ids)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/agent/agent.py", line 688, in _get_flow_run_metadata
result = self.client.graphql(query)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Kevin Kho
05/24/2022, 2:53 AMDylan Lim
05/24/2022, 2:56 AMKevin Kho
05/24/2022, 3:00 AMDylan Lim
05/24/2022, 3:04 AMdef build_flow():
pwd = PrefectSecret("PASSWORD")
with Flow("etl") as flow:
data = extract()
values = transform(data)
output = load(values, pwd)
return flow
if __name__ == "__main__":
flow = build_flow()
flow.register(project_name="test_runs")
I tested another flow with a similar structure yesterday and it was successful; didn’t encounter this error. I also just tried reauthenticating but ran into the same error message in the agent logs.
If it helps - the extract
task pulls data from an API, the values
task parses the response and returns a list of tuples, and the load
task uses PostgresExecuteMany
to write the list of values into a dbKevin Kho
05/24/2022, 3:08 AMDylan Lim
05/24/2022, 3:11 AMdef build_flow():
with Flow("etl_renamed") as flow:
pwd = PrefectSecret("PASSWORD")
data = extract()
values = transform(data)
output = load(values, pwd)
return flow
if __name__ == "__main__":
flow = build_flow()
flow.register(project_name="test_runs")
Kevin Kho
05/24/2022, 3:22 AMyesterday = (datetime.date.today() - datetime.timedelta(days=1)).strftime('%Y-%m-%d')
will be fixed once because it’s calculated during registration time. This should be a task to calculate it every time you run the flow.
What is the Prefect version of registration and of the agent?Dylan Lim
05/24/2022, 3:35 AMKevin Kho
05/24/2022, 3:40 AMpip show prefect
Dylan Lim
05/24/2022, 3:42 AMName: prefect
Version: 1.2.1
Summary: The Prefect Core automation and scheduling engine.
Home-page: <https://www.github.com/PrefectHQ/prefect>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires: click, cloudpickle, croniter, dask, distributed, docker, importlib-resources, marshmallow, marshmallow-oneofschema, msgpack, mypy-extensions, packaging, pendulum, python-box, python-dateutil, python-slugify, pytz, pyyaml, requests, tabulate, toml, urllib3
Required-by:
Kevin Kho
05/24/2022, 3:44 AMDylan Lim
05/24/2022, 3:45 AMAnna Geller
05/24/2022, 10:51 AMDylan Lim
05/24/2022, 11:07 AMAnna Geller
05/24/2022, 11:38 AMfrom prefect import task, Flow
with Flow("etl") as flow:
pwd = PrefectSecret("PASSWORD")
data = extract()
values = transform(data)
output = load(values, pwd)
then from CLI:
prefect register --project test_runs -p yourflow.py
Dylan Lim
05/24/2022, 3:01 PMprefect agent local start
doesn’t work on my machineAnna Geller
05/25/2022, 12:31 PMKevin Kho
05/25/2022, 2:42 PMDylan Lim
05/26/2022, 2:30 AMKevin Kho
05/26/2022, 2:35 AMDylan Lim
05/26/2022, 2:49 AMKevin Kho
05/26/2022, 3:02 AMDylan Lim
05/26/2022, 3:13 AMflow.run()
works but flow.register(...)
+ prefect agent local start
doesn’t and produces the Failed to query for flow run metadata
errorKevin Kho
05/26/2022, 3:14 AMprefect diagnostics
?Dylan Lim
05/26/2022, 3:14 AM{
"config_overrides": {},
"env_vars": [],
"system_information": {
"platform": "macOS-12.3.1-x86_64-i386-64bit",
"prefect_backend": "cloud",
"prefect_version": "1.2.2",
"python_version": "3.9.2"
}
}
Kevin Kho
05/26/2022, 3:16 AMprefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Dylan Lim
05/26/2022, 3:19 AMKevin Kho
05/26/2022, 2:32 PMconfig.toml
though to be sure you are using all the defaults.