Hey We re getting a lot of these errors on our K8s agent and

Question

Hey We re getting a lot of these errors on our K8s agent and it s preventing the server and agent communicating as expected Any quick fixes for this Error in thread thread

James Keegan · Answer

```` 2022 06 07 13 38 40 262 ERROR agent | Error while managing existing k8s jobs Traceback most recent call last File usr local lib python3 6 site packages prefect agent kubernetes agent py line 409 in heartbeat self manage jobs File usr local lib python3 6 site packages prefect agent kubernetes agent py line 349 in manage jobs level= ERROR File usr local lib python3 6 site packages prefect client client py line 2062 in write run logs mutation variables=dict input=dict logs=logs File usr local lib python3 6 site packages prefect client client py line 569 in graphql raise ClientError result errors prefect exceptions ClientError message Foreign key violation locations line 2 column 5 path write run logs extensions code INTERNAL SERVER ERROR exception message Foreign key violation ERROR agent Error while managing existing k8s jobs```

Anna Geller · Answer

what s your `prefect diagnostics`

Anna Geller · Answer

looks like some issue with duplicated or deleted flow runs could be that backend tries to write logs for a flow run that got deleted no longer exists hard to say based only on that info you gave so far

James Keegan · Answer

``` config overrides server database connection url true ui apollo url true env vars PREFECT CONTEXT SECRETS SP USER PREFECT CONTEXT SECRETS TABLEAU USERNAME PREFECT CONTEXT SECRETS RFL SUGARCRM DB HOST PREFECT CONTEXT SECRETS PRO14 DD API KEY PREFECT CONTEXT SECRETS FS AUTH PASSWORD PREFECT CONTEXT SECRETS SP PASS PREFECT CONTEXT SECRETS RFL SUGARCRM DB PORT PREFECT CONTEXT SECRETS RFL SUGARCRM DB USER PREFECT CONTEXT SECRETS TABLEAU PASSWORD PREFECT CONTEXT SECRETS INCROWD SFTP EMBULK USER PREFECT CONTEXT SECRETS DREMIO DB USER PREFECT CONTEXT SECRETS SP DB PREFECT CONTEXT SECRETS INCROWD SFTP EMBULK PASS PREFECT CONTEXT SECRETS SP HOST PREFECT CONTEXT SECRETS DREMIO DB PORT PREFECT CONTEXT SECRETS SP PORT PREFECT CONTEXT SECRETS RFL SUGARCRM DB DB PREFECT CONTEXT SECRETS DREMIO DB PASS PREFECT CONTEXT SECRETS RFL SUGARCRM DB PASS PREFECT CONTEXT SECRETS DREMIO DB HOST PREFECT CONTEXT SECRETS TABLEAU HOST PREFECT CONTEXT SECRETS FS AUTH USERNAME PREFECT CONTEXT SECRETS INCROWD SFTP EMBULK HOST PREFECT CONTEXT SECRETS INCROWD SFTP EMBULK PORT system information platform Linux 5 4 0 110 generic x86 64 with glibc2 31 prefect backend server prefect version 1 2 0 python version 3 9 7 ``` Diagnostics point up

James Keegan · Answer

If you need anything else then let me know thumbsup restarting the server didn t have any affect either will see if I can restart the externalised Postgres cluster

James Keegan · Answer

No luck so far here s an output from the server s logs

James Keegan · Answer

```graphql 1 | graphql 1 | During handling of the above exception another exception occurred graphql 1 | graphql 1 | Traceback most recent call last graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 668 in await completed graphql 1 | return await completed graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 653 in await result graphql 1 | return type field nodes info path await result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 727 in complete value graphql 1 | raise result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 626 in await result graphql 1 | return await result graphql 1 | File prefect server src prefect server graphql extensions py line 52 in resolve graphql 1 | result = await result graphql 1 | File prefect server src prefect server graphql logs py line 26 in resolve write run logs graphql 1 | await api logs create logs logs graphql 1 | File prefect server src prefect server api logs py line 47 in create logs graphql 1 | await models Log insert many model logs selection set= affected rows graphql 1 | File prefect server src prefect server database orm py line 318 in insert many graphql 1 | insert mutation name=cls root fields get insert graphql 1 | File prefect server src prefect server database hasura py line 237 in insert graphql 1 | result = await self execute mutations in transaction mutations= graphql graphql 1 | File prefect server src prefect server database hasura py line 165 in execute mutations in transaction graphql 1 | as box=as box graphql 1 | File prefect server src prefect server database hasura py line 91 in execute graphql 1 | raise ValueError Foreign key violation graphql 1 | ValueError Foreign key violation graphql 1 | graphql 1 | The above exception was the direct cause of the following exception graphql 1 | graphql 1 | Traceback most recent call last graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 668 in await completed graphql 1 | return await completed graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 653 in await result graphql 1 | return type field nodes info path await result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 727 in complete value graphql 1 | raise result graphql 1 | File usr local lib python3 7 site packages graphql execution execute py line 626 in await result graphql 1 | return await result graphql 1 | File prefect server src prefect server graphql extensions py line 52 in resolve graphql 1 | result = await result graphql 1 | File prefect server src prefect server graphql logs py line 26 in resolve write run logs graphql 1 | await api logs create logs logs graphql 1 | File prefect server src prefect server api logs py line 47 in create logs graphql 1 | await models Log insert many model logs selection set= affected rows graphql 1 | File prefect server src prefect server database orm py line 318 in insert many graphql 1 | insert mutation name=cls root fields get insert graphql 1 | File prefect server src prefect server database hasura py line 237 in insert graphql 1 | result = await self execute mutations in transaction mutations= graphql graphql 1 | File prefect server src prefect server database hasura py line 165 in execute mutations in transaction graphql 1 | as box=as box graphql 1 | File prefect server src prefect server database hasura py line 91 in execute graphql 1 | raise ValueError Foreign key violation graphql 1 | graphql error graphql error GraphQLError Foreign key violation graphql 1 | graphql 1 | GraphQL request 2 3 graphql 1 | 1 | mutation $input write run logs input graphql 1 | 2 | write run logs input $input graphql 1 | | ^ graphql 1 | 3 | success graphql 1 | INFO 172 19 0 4 46026 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46036 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46034 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46042 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46044 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46050 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46058 POST HTTP 1 1 200 OK graphql 1 | INFO 172 19 0 4 46060 POST HTTP 1 1 200 OK```

Anna Geller · Answer

can you try adding this env variable could be a Hasura issue ```export HASURA GRAPHQL V1 BOOLEAN NULL COLLAPSE=true``` more on that <https discourse prefect io t we upgrade from 0 15 14 to 1 1 0 and it seems like our flows are still running but nothing shows up on the dashboard unless we pick a project 825|https discourse prefect io t we upgrade from 0 15 14 to 1 1 0 and it seems like our f ing shows up on the dashboard unless we pick a project 825>