Hi team our team has recently deployed prefect on gcp kubern

Question

Hi team our team has recently deployed prefect on gcp kubernete and it keeps saying `Couldn t connect to Prefect Server at <http 34 82 119 23 4200 graphql>` can anyone help

Kevin Kho · Answer

Is this message when you go to the UI

Dave · Answer

yeah

Dave · Answer

this error only occurs after recent prefect UI change

Kevin Kho · Answer

There doesn t seem to be any <https github com PrefectHQ ui blob master Changelog md 2022 01 14|change> that would be related What was your previous version and current version

Dave · Answer

UI Release Jan 14 6 50AM

Kevin Kho · Answer

What was the previous Is your API pod healthy

Dave · Answer

all status seems good

Dave · Answer

the same flow was running well until yesterday

Kevin Kho · Answer

Am a bit confused I thought the error was from the UI but your UI is loading

Kevin Kho · Answer

Ah I see

Kevin Kho · Answer

Are your agents still running Able to hit the API

Dave · Answer

what agents and how to hit the api

Dave · Answer

sry Im kinda new

Kevin Kho · Answer

The agents are the things that run the Flows so you must have had some online at some point Actually ignore that Could you try going to the GraphQL playground at <http 34 82 119 23 4200>

Dave · Answer

yes I can

Kevin Kho · Answer

Could you try the hello world query ```query hello ```

Dave · Answer

only got error msg

Dave · Answer

it says `connect ECONNREFUSED 10 132 14 47 4201`

Kevin Kho · Answer

There looks to be something down in the backend I don t think this is a UI issue Could you check the Hasura and GraphQL pod for logs And I guess might as well check the database is good These are all in the made k8s cluster right

Kevin Kho · Answer

Just asking on the cluster because someone had a CORS issue recently but they were doing cross cluster things

Dave · Answer

`Could not upgrade the database `

Kevin Kho · Answer

Maybe try running the database migration manually from the command <https github com PrefectHQ server running the system as a developer|here>

Dave · Answer

where should I run these command

Zanie · Answer

It looks like your GraphQL service isn t starting because the init container is failing to run migrations Are there logs in your hasura pod

Kevin Kho · Answer

One sec let me ask someone The answer is the graphql pod but it might not help you

Dave · Answer

Sure take your time

Zanie · Answer

I see some info level logs without any helpful errors in that screenshot hm

Zanie · Answer

What helm chart version are you on Hasura was recently upgraded

Dave · Answer

where can i check helm chart version

Zanie · Answer

`helm list` should do it

Dave · Answer

hmm I m on GCP and where should I got to run the command

Zanie · Answer

Ah from a terminal you can use the gcloud cli to configure kubectl to connect correctly

Zanie · Answer

There also may be versioning information in the pod labels or something

Dave · Answer

prefect server hasura was created Apr 14 2021

Dave · Answer

apiVersion v1 kind Service metadata annotations ingress true creationTimestamp 2021 04 14T21 09 52Z labels hasura prefect server Helm prefect server latest prefect server 0 0 0 managedFields apiVersion v1 fieldsType FieldsV1 fieldsV1 f metadata f labels f f f f f f f spec f ports k port 3000 protocol TCP f name f port f protocol f targetPort f selector f f f f sessionAffinity f type manager Go http client operation Update time 2021 04 14T21 09 52Z name prefect server hasura namespace default resourceVersion 8964515 uid 47ddfda5 3ac6 49f7 8fd6 c07a818242b1 spec clusterIP 10 132 1 243 clusterIPs 10 132 1 243 ports name http port 3000 protocol TCP targetPort http selector hasura prefect server prefect server sessionAffinity None type ClusterIP status loadBalancer

Dave · Answer

this is the yaml file does it help

Dave · Answer

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prefect server default 8 2021 04 14 16 07 02 31896 0700 0700 deployed prefect server 0 0 0 latest

Zanie · Answer

Ah it looks like you ve installed from GitHub instead of the released helm chart

Dave · Answer

the guy who installed it has left the team joy

Zanie · Answer

I see haha it looks like a pretty old version but that shouldn t matter

Zanie · Answer

What is the image tag for the hasura pod

Zanie · Answer

It seems likely that since you re on the latest release instead of a version pin your GraphQL container has been updated to the newest version which is compatible with Hasura 2 x but your Hasura container is still running 1 x

Dave · Answer

so should I update Hasura

Dave · Answer

im wondering why it happends these two days

Dave · Answer

actually there was no issue before that

Zanie · Answer

If you pod rolled over it could grab the latest image

Zanie · Answer

I d recommend installing the helm chart from a release slightly smiling face

Dave · Answer

will it impact anything

Dave · Answer

and how to install

Zanie · Answer

I m not sure how to do the upgrade without affecting your current services it depends on how your chart was deployed in the first place

Zanie · Answer

Here s the README for installation <https github com PrefectHQ server tree master helm prefect server installing released versions>

Zanie · Answer

Basically You should be fine if you ensure the same database is used but I m not sure how your database was configured and don t want to try to give advice from over here

Zanie · Answer

Whoever installed your chart did so from the development version which is unfortunate

Dave · Answer

do you think the problem can be solved by restalling the helm chart

Kevin Kho · Answer

Hopefully cuz your versions across the services would now be compatible

Dave · Answer

Thank you Michael and Kevin for helping I will let you know if have any issue while installing

Sankalp Patnaik · Answer

How can I find the config yaml or value yaml file location if I was not present at the time of installation

Kevin Kho · Answer

Do you mean `config toml` What are you trying to do

Sankalp Patnaik · Answer

yes Kevin I was trying to upgrade helm chart but it failed with Error UPGRADE FAILED rendered manifests contain a resource that already exists Unable to continue with update RoleBinding prefect server agent role binding in namespace default exists and cannot be imported into the current release invalid ownership metadata annotation validation error missing key must be set to prefect server annotation validation error missing key must be set to default I guess we may need to check the toml file

Sankalp Patnaik · Answer

The error we are getting in GKE workload

Kevin Kho · Answer

Ah my bad I see what you are saying The values yaml comes from <https github com PrefectHQ server tree master helm prefect server|here> but I don t know hot to get the `config yaml` that was used Because it s something you have to continue to provide

Sankalp Patnaik · Answer

I was trying to get more information so i checked the logs of pod and got attached error So now what can be the resolution

Sankalp Patnaik · Answer

Error applying Hasura metadata from prefect server services hasura migrations metadata yaml Could not upgrade the database Error Failed to apply Hasura metadata Error 400 Client Error Bad Request for url <http prefect server hasura default 3000 v1 query>

Kevin Kho · Answer

I d have to ask Michael when he s around

Dave · Answer

Hi Kevin is Michael around

Zanie · Answer

Hey Dave is this after upgrading the chart to a pinned version

Zanie · Answer

You can also try updating the tags on the containers from latest to an older version

Zanie · Answer

Anything before we upgraded Hasura should work

Dave · Answer

Hey Michael actually I meet such issue when update helm

Zanie · Answer

Looks like you didn t run `helm repo update`

Dave · Answer

lemme try it

Zanie · Answer

You re probably better off pinning those containers to an old version rather than upgrading though

Zanie · Answer

If you don t know how it s setup you could delete your database

Dave · Answer

how to pin those containers to an older version

Zanie · Answer

You ll need to change the tag specified in the deployment from latest to a version

Zanie · Answer

Dave · Answer

can i directly change it in yaml file

Zanie · Answer

Yeah if GCP supports that But you want to change the image tag not the label

Dave · Answer

sorry I m still confused can we have a video call if you dont mind

Zanie · Answer

I cannot You ll have to go through < George Coyne>

Dave · Answer

alright no problem

Dave · Answer

how can I change the image tag

Kevin Kho · Answer

unless Michael corrects me I think you should see a containers section there with the graphql container and this will have an image Hasura was upgraded to 2 0 and this likely broke the setup so you need to pin all of the Prefect to something like over a month ago Maybe the `core 0 15 11` tag for example for the image You then need to do this for all the Prefect containers For graphql for example you would replace line

George Coyne · Answer

We just hopped off of a webex we think we found the values yml that was used to deploy the current instance and he knows where to make tag changes

Dave · Answer

Hey do we need to re deploy after making the tag changes

Kevin Kho · Answer

I would think so

Sankalp Patnaik · Answer

Hi < Kevin Kho> < Zanie> < George Coyne> I redeployed the helm with a different name as prefect serverr previously it was prefect server

Sankalp Patnaik · Answer

I was succefull in deploting and in kubernetics engine I can see those pods are running But the UI is still not active same error in UI How can I resolve this

Kevin Kho · Answer

Hi < Sankalp Patnaik> no need to tag multiple team members like that It s hard to say what is going on here Do you have some additional setup beyond the standard helm chart

Sankalp Patnaik · Answer

Yes Kevin we are using Docker to buils an image where I have added agent local and prefect server start apart from this Kubernetics to spin the pod

Kevin Kho · Answer

There are some things that look off here 1 You shoudn t need to call `prefect server start` if you are already spinning up server using helm `prefect server start` spins up the services as Docker containers with `docker compose` while using the helm spins up the services and Kubernetes deployments They are two ways to deploy and you don t need both 2 I don t think you should be calling `prefect server start` inside a Dockerfile Again `prefect server start` uses Docker compose so this will run docker in docker It s not intended to be run inside a container

Sankalp Patnaik · Answer

I have one doubt Kevin in below image Prefect server helm was created last year which stopped working Prefect serverr helm 2nd one highlighted created by me last week which was able to spin pods in GKE Should I create the helm chart with Prefect server only or any name should work I fear I am using an additional r highlighted at athe end which is causing problems

Kevin Kho · Answer

Per the <https github com PrefectHQ server tree master helm prefect server|readme> under number 4 I think it should be fine The previous installation you had of server seems to have been against the master branch like the original issue with Dave here The setup you are doing should be more stable

Sankalp Patnaik · Answer

Hi < Kevin Kho> I used `prefect server config` to generate the Docker Compose file and then `docker compose up` to spin up the server s services When I tried to start the Agent I got the following error prefect utilities exceptions ClientError message request to failed reason connect ECONNREFUSED 10 132 1 243 3000 locations line 2 column 5 path project extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 132 1 243 3000 type system errno ECONNREFUSED code ECONNREFUSED

Sankalp Patnaik · Answer

am i missing something while spinning in docker