Hi all, I am on

0.13.10

prefect server

, and I started to see this error for scheduled task and manually run tasks

[{'message': 'Foreign key violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_flow_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Foreign key violation.'}}}]

It seems like it comes from the graphql side of things. But I couldn’t figure out why it randomly fails with that error. After a couple of manual restart sometimes it works. I tried to debug but since it happens at the very beginning of the flow, there’s not much that I can find out When I register the flow I register it with the same core version, or at least I try to

Hi @Keat - can you describe what you're (with code if possible) when you say you're scheduling and manually running tasks?

@nicholas I am sorry, I always try to give as much context as possible but I still failed at it. Maybe I just like having you tag my name and ask me more questions. What I meant was the flow run that are scheduled to run and flow runs that I manually clicked run would randomly get that error. One suspect I currently have is, I have another virtual environment that has a prefect core version that is different from that server and I registered a flow from there that might kickstart this error. But since this is affecting my previous flow registered with the same

core

version, I ruled them out, for now. Another one is I restarted my local agent and provide new env var to the agent. But I am not sure where/how to start debugging with just that error message from gql

No worries at all! That foreign key violation is usually a result of trying to write flow run states for a flow run that no longer exists, does that sound plausible?

Hmm, removed another project that is unrelated to the flow, but probably good enough opening of a pigeon hole to get me going at it over the weekend. Thanks again nicholas ! Will report back !

Sounds good @Keat, keep me posted 😄

Okay apparently there are logs from postgres end.

postgres_1  | 2020-10-23 16:37:55.136 UTC [28323] ERROR:  insert or update on table "flow_run" violates foreign key constraint "flow_run_agent_id_fkey"
postgres_1  | 2020-10-23 16:37:55.136 UTC [28323] DETAIL:  Key (agent_id)=(fe47d130-3aa1-4fda-93c4-509c3f911497) is not present in table "agent".
postgres_1  | 2020-10-23 16:37:55.136 UTC [28323] STATEMENT:  WITH "flow_run__mutation_result_alias" AS (UPDATE "public"."flow_run" SET "agent_id" = ($1)::uuid  WHERE (('true') AND ((((("public"."flow_run"."id") = (($2)::uuid)) AND ('true')) AND ('true')) AND ('true'))) RETURNING * , CASE WHEN 'true' THEN NULL ELSE "hdb_catalog"."check_violation"('update check constraint failed')  END ), "flow_run__all_columns_alias" AS (SELECT  "id" , "tenant_id" , "created" , "flow_id" , "parameters" , "scheduled_start_time" , "auto_scheduled" , "heartbeat" , "start_time" , "end_time" , "version" , "state" , "state_timestamp" , "state_message" , "state_result" , "state_start_time" , "serialized_state" , "name" , "context" , "times_resurrected" , "updated" , "idempotency_key" , "agent_id"  FROM "flow_run__mutation_result_alias"      ) SELECT  json_build_object('affected_rows', (SELECT  COUNT(*)  FROM "flow_run__all_columns_alias"      ) )

Current agent_id is

4edb33ee-a064-4cc4-8c4b-212cef7962bd

. So it could be me starting and stopping the agent via

ctrl+c

and also from the

UI

. When I query the agent table via the api, there’s only the current agent running but not the

id

listed in this error.

Ahah! In which case we have a ticket open for that 🙂 https://github.com/PrefectHQ/server/issues/114

Damn, he managed to pin-point to the root cause of the bug so well. Thanks, I shall also browse issues next time.

No worries at all, thanks for reporting it 🙂