Hello! I’m trying to run a registered flow but I got this error message in the agent logs, could anyone help to explain what this error means?

Is this Prefect Cloud or Server?

I’m using cloud. I tried removing it from the cloud UI and re registering but still got the same error. The flow on the cloud ui just shows that it's pending, not even failing

Is the Flow simple enough to share?

Sure, let me abstract some of the details out:

def build_flow():
    pwd = PrefectSecret("PASSWORD")

    with Flow("etl") as flow:
        data = extract()
        values = transform(data)
        output = load(values, pwd)
    return flow

if __name__ == "__main__":
    flow = build_flow()
    flow.register(project_name="test_runs")

I tested another flow with a similar structure yesterday and it was successful; didn’t encounter this error. I also just tried reauthenticating but ran into the same error message in the agent logs. If it helps - the

extract

task pulls data from an API, the

values

task parses the response and returns a list of tuples, and the

load

task uses

PostgresExecuteMany

to write the list of values into a db

The only thing that looks off is the secret outside of the Flow. But I think that should error when it goes to the load task, not at the flow start. Could you try maybe: 1. move secret to inside 2. register the flow under a different name to test?

Just tried both - sadly I’m still getting the same error messages 😞

def build_flow():
    with Flow("etl_renamed") as flow:
        pwd = PrefectSecret("PASSWORD")

        data = extract()
        values = transform(data)
        output = load(values, pwd)
    return flow

if __name__ == "__main__":
    flow = build_flow()
    flow.register(project_name="test_runs")

That’s very weird, Could you DM me the whole flow?

ah thanks for the spot! Will put that in a task 🙂 For the version of registration, how can I retrieve it? I deleted the previous flow and re-registered so it’s currently just version 1. For the agent, I’m using local

More like

pip show prefect

oh right 🤦‍♂️ here it is:

Name: prefect
Version: 1.2.1
Summary: The Prefect Core automation and scheduling engine.
Home-page: <https://www.github.com/PrefectHQ/prefect>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires: click, cloudpickle, croniter, dask, distributed, docker, importlib-resources, marshmallow, marshmallow-oneofschema, msgpack, mypy-extensions, packaging, pendulum, python-box, python-dateutil, python-slugify, pytz, pyyaml, requests, tabulate, toml, urllib3
Required-by:

I can’t see anything wrong immediately. I’ll leave a message for the team and get back to you tom

Thanks! Appreciate it @Kevin Kho

You need to move the flow object creation out of main. Currently the flow object is only created when you run flow locally

Hey @Anna Geller! Thanks for the response - I tried removing the main, creating a new project and re-registering and re-running the agent; but I still goto the same error

try this:

from prefect import task, Flow


with Flow("etl") as flow:
    pwd = PrefectSecret("PASSWORD")
    data = extract()
    values = transform(data)
    output = load(values, pwd)

then from CLI:

prefect register --project test_runs -p yourflow.py

Hi @Anna Geller - I’ve tried to do this but the same error message pops up. I also tried refreshing API keys and logging out / relogging in and restarting agents/projects, stripping down to the bare minimum (eg hello world flow), which led to some weird behaviour - two out of 15 or so flows mysteriously triggered but the rest gave the same error messages. Still can’t really find a pattern, so I’ll just sleep on it for now…

"X doesn't work" is a bit of an oversimplification giving us very little info to help 😄 can you say more? even better - could you try writing up some README GitHub Gist documenting step by step what you have done so far, what were the results of it and which behavior was surprising to you or didn't work as expected? this would significantly help to see a bigger picture and identify what might be happening

That’s very weird. But it’s only this flow that fails also right? On a specific machine?

ah sorry - by doesn’t work I mean that I get the same error messages in this thread (it’s been the same error since the start) And @Kevin Kho strangely it’s all flows; I even opened a new Prefect account and logged in but the same error message shows up. However, when I ssh into my remote server and register flows from there + run the agent there, things work fine 😅 It’s probably something funky on my machine that somehow caused the error

Do you have another machine you can try too?

Not at the moment, but I’ll see if I can test it on a friend’s laptop. Just a hunch, but does deleting projects / flows and re-creating them with the same name cause problems?

Not problems like the one you are facing. Deleting projects is a soft delete so it’s marked for deletion and then actual deletion happens like 15 minutes later maybe. If you try to delete and create immediately, you will get an error saying that the project name already exists. The error you have is bizarre for me. Does it happen for a simple “hello world” flow too?

@Kevin Kho yep unfortunately 😞 it happens for all flows registered from my machine. To clarify,

flow.run()

works but

flow.register(...)

+

prefect agent local start

doesn’t and produces the

Failed to query for flow run metadata

error

Can you run

prefect diagnostics

?

For now, my workaround is to develop and push my code from my laptop but register it and run the agent on our server

Do you have anything in the config.toml or env var settings?

Yeah - spinning up the agent on my machine seems to always cause this error. I’ll dm you the config.toml contents

I looked at it and it looked pretty fine. Could not identify anything wrong. You can also try an empty

config.toml

though to be sure you are using all the defaults.