https://prefect.io logo
Join Slack
Powered by
Hello Team! Im facing some new errors on actual pr...
# ask-community
r
Hello Team! Im facing some new errors on actual production flows that I´ve never seen, we have had to reboot the server this morning and now we see in two flows thesame error….
Copy code
[{'message': 'Foreign key violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_flow_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Foreign key violation.'}}}]
a
@Raúl Mansilla looks like rebooting the server caused some GraphQL issues when the 
set_flow_run_states
mutation was used. Do you happen to use this mutation yourself in a flow?
INTERNAL_SERVER_ERROR is really only telling us that some GraphQL API call failed. Can you share more about: • can you confirm GraphQL is reachable? • can you look at the server and agent logs to check what might be unusual? • did all flows started failing or only the one you just mentioned?
upvote 1
@Raúl Mansilla I looked at it a bit more and It looks like some flow or agent is trying to set flow run state on a flow run that no longer exists. When you rebooted the server, did you persist all your data? If you didn’t, it’s possible that a flow run that didn’t complete before reboot is trying to finish, but after the reboot this flow run no longer exists.
r
Sorry @Anna Geller I had to get out the office
I´m going to take a deeper look
the last thing you mention could be the problem
👍 1
the data should be persistent but maybe these flows where corrupted somehow, the server was irresponsive so didn´t have other options to recover it than reboot
I started prefect like this -> 
prefect server start --use-volume --volume-path ~/.prefect/pg_data
👍 1
Hi again, graphQL logs says sometimes this:
Copy code
Foreign key violation.

GraphQL request:2:3
1 | mutation ($input: set_flow_run_states_input!) {
2 |   set_flow_run_states(input: $input) {
  |   ^
3 |     states {
Traceback (most recent call last):
  File "/prefect-server/src/prefect_server/database/hasura.py", line 85, in execute
    as_box=as_box,
  File "/prefect-server/src/prefect_server/utilities/graphql.py", line 80, in execute
    raise ValueError(result["errors"])
ValueError: [{'extensions': {'path': '$', 'code': 'constraint-violation'}, 'message': 'Foreign key violation. insert or update on table "flow_run" violates foreign key constraint "flow_run_agent_id_fkey"'}]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
    return await result
  File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
    result = await result
  File "/prefect-server/src/prefect_server/graphql/states.py", line 45, in resolve_set_flow_run_states
    *[check_size_and_set_state(state_input) for state_input in input["states"]]
  File "/prefect-server/src/prefect_server/graphql/states.py", line 39, in check_size_and_set_state
    agent_id=agent_id,
  File "/prefect-server/src/prefect_server/api/states.py", line 108, in set_flow_run_state
    await api.runs.update_flow_run_agent(flow_run_id=flow_run_id, agent_id=agent_id)
  File "/prefect-server/src/prefect_server/api/runs.py", line 410, in update_flow_run_agent
    set={"agent_id": agent_id}
  File "/prefect-server/src/prefect_server/database/orm.py", line 406, in update
    run_mutation=run_mutation,
  File "/prefect-server/src/prefect_server/database/hasura.py", line 391, in update
    result = await self.execute_mutations_in_transaction(mutations=[graphql])
  File "/prefect-server/src/prefect_server/database/hasura.py", line 165, in execute_mutations_in_transaction
    as_box=as_box,
  File "/prefect-server/src/prefect_server/database/hasura.py", line 91, in execute
    raise ValueError("Foreign key violation.")
ValueError: Foreign key violation.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 674, in await_completed
    return await completed
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 659, in await_result
    return_type, field_nodes, info, path, await result
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
    raise result
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
    return await result
  File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
    result = await result
  File "/prefect-server/src/prefect_server/graphql/states.py", line 45, in resolve_set_flow_run_states
    *[check_size_and_set_state(state_input) for state_input in input["states"]]
  File "/prefect-server/src/prefect_server/graphql/states.py", line 39, in check_size_and_set_state
    agent_id=agent_id,
  File "/prefect-server/src/prefect_server/api/states.py", line 108, in set_flow_run_state
    await api.runs.update_flow_run_agent(flow_run_id=flow_run_id, agent_id=agent_id)
  File "/prefect-server/src/prefect_server/api/runs.py", line 410, in update_flow_run_agent
    set={"agent_id": agent_id}
  File "/prefect-server/src/prefect_server/database/orm.py", line 406, in update
    run_mutation=run_mutation,
  File "/prefect-server/src/prefect_server/database/hasura.py", line 391, in update
    result = await self.execute_mutations_in_transaction(mutations=[graphql])
  File "/prefect-server/src/prefect_server/database/hasura.py", line 165, in execute_mutations_in_transaction
    as_box=as_box,
  File "/prefect-server/src/prefect_server/database/hasura.py", line 91, in execute
    raise ValueError("Foreign key violation.")
graphql.error.graphql_error.GraphQLError: Foreign key violation.

GraphQL request:2:3
1 | mutation ($input: set_flow_run_states_input!) {
2 |   set_flow_run_states(input: $input) {
  |   ^
3 |     states {
which is the error that appears in the gui
I´m trying to catch if the server throws something interesting
k
Could you check if your pods are healthy? Could it be something didn’t start right?
r
in this environment we´re using no pods just prefect onto ec2 and a dask cluster in several ec2 and we “rebooted” the cluster as well
k
Oh sorry I mean the containers from 
prefect server start
. Are they all healthy? I guess you can also try going to the interactive tab in the UI and query the hello route:
Copy code
query {
  hello
}
r
Yes, I think they´re healthy
{ “hello”: “👋” }
k
I don’t know the immediate solution. There was a previous thread here . Does this persist for new flow runs?
flow_run_agent_id_fkey
is your constraint. Maybe a new agent will help?
r
Hello
finally, it was a silly thing
there were two instances of two agents running at the same time! thanks for your help @Anna Geller and @Kevin Kho I always learn something new from you
👍 1
k
Oh nice to know. Thanks for circling back on that.
r
👍
3 Views
Open in Slack
PreviousNext
Powered by