been seeing this sporadically over the last 24 hours maybe 1 Prefect Community #ask-community

been seeing this sporadically over the last 24 hou...

Brett Naul

11/03/2021, 12:48 PM

been seeing this sporadically over the last 24 hours (maybe 1 in 10 flows?), any idea what might be up? I don't see anything about

edge_on_conflict

in the

prefect

server

repos

Copy code

File "/src/util/prefect.py", line 575, in register_flow
                                             flow_id = client.register(flow, project_name, idempotency_key=flow.serialized_hash())
                                           File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1227, in register
                                             self.graphql(
                                           File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 569, in graphql
                                             raise ClientError(result["errors"])
                                         prefect.exceptions.ClientError: [***'path': ['register_edges'], 'message': '[***\'extensions\': ***\'path\': \'$.variableValues\', \'code\': \'validation-failed\'***, \'message\': "no such type exists in the schema: \'edge_on_conflict\'"***]', 'extensions': ***'code': 'INTERNAL_SERVER_ERROR'***]

Anna Geller

11/03/2021, 12:53 PM

At first glance, it seems like a problem with registration due to Python/Prefect version mismatch. Can you share more about your setup? 1. Do you run Prefect Server or Cloud? it looks like Server but I’d rather check 2. What Python and Prefect version do you run on your agent vs. on the machine from which do you register your flows - do they match? 3. What Storage do you use? This also plays a role in registration, as some storage classes require a build upon registration

Brett Naul

11/03/2021, 12:55 PM

cloud, GCS storage, k8s agent; and at least some of these errors come from flows registered from inside the same docker container that is then trying to run it, so no version mismatch there..but it could be that the previous version of the flow (registered who knows when) is incompatible somehow?

Anna Geller

11/03/2021, 1:05 PM

Thanks. It looks like it may be related to the API key (or API token in the older versions). If you use a legacy RUNNER-scoped token for registering your flow, you can try to create an API key in the UI, logout and then log in again with the new key, and then reregister this flow:

Copy code

prefect auth logout
prefect auth login --key XXX
prefect register --project YOUR_PROJECT -p /path/to/flow.py

upvote 1

Brett Naul

11/03/2021, 1:17 PM

hm interesting...we definitely have a mix of old and new tokens/keys, I would be a bit surprised if this was the issue though since like I said this is only failing intermittently. the exact same container will succeed at registering+running one flow but not another

Anna Geller

11/03/2021, 1:49 PM

got it, still it would be great if you could try it and report back

Zanie

11/03/2021, 3:21 PM

Are you registering all of your flows the same way? CLI vs

flow.register

? Is it happening to the same flow each time it fails?

Brett Naul

11/03/2021, 3:26 PM

yeah everything is going through the same helper. I was wrong about it being intermittent though: for two very similar flows, one fails and one succeeds. another wrinkle is this

register

call is happening inside another flow also running in cloud, so there's kind of a lot going on. my best guess is that because the succeeding flow is new, something about its state is different from the old one and avoids running into whatever error we're seeing

Zanie

11/03/2021, 3:27 PM

Are you using the CLI in your helper or flow.register() ?

Brett Naul

11/03/2021, 3:58 PM

client.register(flow)

Zanie

11/03/2021, 4:13 PM

We're deploying a possible fix for this right now. We'll let you know when it's live.

🙌 1

Brett Naul

11/03/2021, 4:16 PM

awesome, thanks! 🤞

Brett Naul

11/03/2021, 4:37 PM

the best idea I could come up with so far was to re-register the flow in a new project and/or with a random

idempotency_key

to make extra sure I was getting a new version of the flow and not an old one...no luck though, same error. I also put a print statement right before the

register

call and for the old flow it shows up as a very long API token and the new one as the newer API key. exact same docker image and code path! only the flow is different. very strange

Zanie

11/04/2021, 12:18 AM

This fix should be live now.

4 Views

Open in Slack

Previous Next