https://prefect.io logo
d

Dave

01/31/2022, 8:21 PM
Hi team, our team has recently deployed prefect on gcp kubernete. and it keeps saying
Couldn't connect to *Prefect Server* at *<http://34.82.119.23:4200/graphql>
,* can anyone help?
k

Kevin Kho

01/31/2022, 8:23 PM
Is this message when you go to the UI?
d

Dave

01/31/2022, 8:23 PM
yeah
this error only occurs after recent prefect UI change
k

Kevin Kho

01/31/2022, 8:26 PM
There doesn’t seem to be any change that would be related. What was your previous version and current version?
d

Dave

01/31/2022, 8:26 PM
UI Release Jan 14 • 6:50AM
k

Kevin Kho

01/31/2022, 8:28 PM
What was the previous? Is your API pod healthy?
d

Dave

01/31/2022, 8:28 PM
all status seems good
the same flow was running well until yesterday
k

Kevin Kho

01/31/2022, 8:32 PM
Am a bit confused. I thought the error was from the UI but your UI is loading?
d

Dave

01/31/2022, 8:32 PM
k

Kevin Kho

01/31/2022, 8:32 PM
Ah I see
Are your agents still running? Able to hit the API?
d

Dave

01/31/2022, 8:34 PM
what agents? and how to hit the api?
sry Im kinda new
k

Kevin Kho

01/31/2022, 8:37 PM
The agents are the things that run the Flows so you must have had some online at some point. Actually ignore that. Could you try going to the GraphQL playground at http://34.82.119.23:4200?
d

Dave

01/31/2022, 8:37 PM
yes I can
k

Kevin Kho

01/31/2022, 8:38 PM
Could you try the hello world query?
Copy code
query {
    hello
}
d

Dave

01/31/2022, 8:39 PM
only got error msg
it says
connect ECONNREFUSED 10.132.14.47:4201
k

Kevin Kho

01/31/2022, 8:43 PM
There looks to be something down in the backend. I don’t think this is a UI issue. Could you check the Hasura and GraphQL pod for logs? And I guess might as well check the database is good? These are all in the made k8s cluster right?
Just asking on the cluster because someone had a CORS issue recently but they were doing cross-cluster things.
d

Dave

01/31/2022, 8:44 PM
Could not upgrade the database!
k

Kevin Kho

01/31/2022, 8:52 PM
Maybe try running the database migration manually from the command here ?
d

Dave

01/31/2022, 8:54 PM
where should I run these command?
z

Zanie

01/31/2022, 8:55 PM
It looks like your GraphQL service isn’t starting because the init container is failing to run migrations. Are there logs in your hasura pod?
k

Kevin Kho

01/31/2022, 8:55 PM
One sec let me ask someone. The answer is the graphql pod but it might not help you
d

Dave

01/31/2022, 8:56 PM
Sure, take your time!
z

Zanie

01/31/2022, 8:57 PM
I see some info level logs without any helpful errors in that screenshot, hm.
What helm chart version are you on? Hasura was recently upgraded.
d

Dave

01/31/2022, 8:59 PM
where can i check helm chart version?
z

Zanie

01/31/2022, 8:59 PM
helm list
should do it
d

Dave

01/31/2022, 9:00 PM
hmm, I'm on GCP and where should I got to run the command?
z

Zanie

01/31/2022, 9:02 PM
Ah from a terminal, you can use the gcloud cli to configure kubectl to connect correctly
There also may be versioning information in the pod labels or something
d

Dave

01/31/2022, 9:04 PM
prefect-server-hasura was created Apr 14, 2021
apiVersion: v1 kind: Service metadata: annotations: cloud.google.com/neg: '{"ingress":true}' creationTimestamp: "2021-04-14T210952Z" labels: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: prefect-server app.kubernetes.io/version: latest helm.sh/chart: prefect-server-0.0.0 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: fmetadata flabels .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:app.kubernetes.io/version: {} f:helm.sh/chart: {} fspec fports .: {} k:{"port":3000,"protocol":"TCP"}: .: {} fname {} fport {} fprotocol {} ftargetPort {} fselector .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/name: {} fsessionAffinity {} ftype {} manager: Go-http-client operation: Update time: "2021-04-14T210952Z" name: prefect-server-hasura namespace: default resourceVersion: "8964515" uid: 47ddfda5-3ac6-49f7-8fd6-c07a818242b1 spec: clusterIP: 10.132.1.243 clusterIPs: - 10.132.1.243 ports: - name: http port: 3000 protocol: TCP targetPort: http selector: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/name: prefect-server sessionAffinity: None type: ClusterIP status: loadBalancer: {}
this is the yaml file, does it help?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prefect-server default 8 2021-04-14 160702.31896 -0700 -0700 deployed prefect-server-0.0.0 latest
z

Zanie

01/31/2022, 9:08 PM
Ah it looks like you’ve installed from GitHub instead of the released helm chart
d

Dave

01/31/2022, 9:09 PM
the guy who installed it has left the team 😂
z

Zanie

01/31/2022, 9:11 PM
I see haha, it looks like a pretty old version but that shouldn’t matter
What is the image tag for the hasura pod?
It seems likely that since you’re on the “latest” release instead of a version pin your GraphQL container has been updated to the newest version which is compatible with Hasura 2.x but your Hasura container is still running 1.x
d

Dave

01/31/2022, 9:13 PM
so should I update Hasura?
im wondering why it happends these two days
actually there was no issue before that
z

Zanie

01/31/2022, 9:14 PM
If you pod rolled over it could grab the latest image
I’d recommend installing the helm chart from a release 🙂
d

Dave

01/31/2022, 9:15 PM
will it impact anything?
and how to install?
z

Zanie

01/31/2022, 9:17 PM
I’m not sure how to do the upgrade without affecting your current services, it depends on how your chart was deployed in the first place.
Basically: You should be fine if you ensure the same database is used, but I’m not sure how your database was configured and don’t want to try to give advice from over here.
Whoever installed your chart did so from the development version, which is unfortunate.
d

Dave

01/31/2022, 9:24 PM
do you think the problem can be solved by restalling the helm chart?
k

Kevin Kho

01/31/2022, 9:39 PM
Hopefully cuz your versions across the services would now be compatible.
d

Dave

01/31/2022, 9:50 PM
Thank you, Michael and Kevin, for helping! I will let you know if have any issue while installing.
s

Sankalp Patnaik

02/01/2022, 6:20 PM
How can I find the config.yaml or value.yaml file location if I was not present at the time of installation
k

Kevin Kho

02/01/2022, 6:22 PM
Do you mean
config.toml
? What are you trying to do?
s

Sankalp Patnaik

02/01/2022, 6:24 PM
yes Kevin,I was trying to upgrade helm chart but it failed with "Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: RoleBinding "prefect-server-agent-role-binding" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "prefect-server"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"" I guess we may need to check the toml file
The error we are getting in GKE workload
k

Kevin Kho

02/01/2022, 6:35 PM
Ah my bad I see what you are saying. The values.yaml comes from here, but I don’t know hot to get the
config.yaml
that was used. Because it’s something you have to continue to provide
s

Sankalp Patnaik

02/02/2022, 8:10 AM
I was trying to get more information so i checked the logs of pod and got attached error.So now what can be the resolution ?
Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml Could not upgrade the database! Error: Failed to apply Hasura metadata! Error: 400 Client Error: Bad Request for url: http://prefect-server-hasura.default:3000/v1/query
k

Kevin Kho

02/02/2022, 2:16 PM
I’d have to ask Michael when he’s around
d

Dave

02/02/2022, 7:14 PM
Hi Kevin, is Michael around?
z

Zanie

02/02/2022, 7:32 PM
Hey Dave, is this after upgrading the chart to a pinned version?
You can also try updating the tags on the containers from “latest” to an older version
Anything before we upgraded Hasura should work
d

Dave

02/02/2022, 7:47 PM
Hey Michael, actually I meet such issue when update helm
z

Zanie

02/02/2022, 7:49 PM
Looks like you didn’t run
helm repo update
d

Dave

02/02/2022, 7:50 PM
lemme try it
z

Zanie

02/02/2022, 7:50 PM
You’re probably better off pinning those containers to an old version rather than upgrading though
If you don’t know how it’s setup, you could delete your database.
d

Dave

02/02/2022, 7:52 PM
how to pin those containers to an older version?
z

Zanie

02/02/2022, 7:56 PM
You’ll need to change the tag specified in the deployment from “latest” to a version
d

Dave

02/02/2022, 8:04 PM
can i directly change it in yaml file?
z

Zanie

02/02/2022, 8:05 PM
Yeah if GCP supports that! But you want to change the image tag not the label.
d

Dave

02/02/2022, 8:07 PM
sorry I'm still confused, can we have a video call if you dont mind?
z

Zanie

02/02/2022, 8:11 PM
I cannot. You’ll have to go through @George Coyne
d

Dave

02/02/2022, 8:12 PM
alright, no problem
how can I change the image tag?
k

Kevin Kho

02/02/2022, 8:40 PM
unless Michael corrects me, I think you should see a containers section there with the graphql container and this will have an image. Hasura was upgraded to 2.0 here and this likely broke the setup so you need to pin all of the Prefect containers to something like over a month ago. Maybe the
core-0.15.11
tag for example for the server image. You then need to do this for all the Prefect containers. For graphql for example, you would replace this line
g

George Coyne

02/02/2022, 8:45 PM
We just hopped off of a webex, we think we found the values.yml that was used to deploy the current instance and he knows where to make tag changes.
d

Dave

02/02/2022, 9:56 PM
Hey, do we need to re-deploy after making the tag changes?
k

Kevin Kho

02/02/2022, 10:09 PM
I would think so
s

Sankalp Patnaik

02/14/2022, 6:25 AM
Hi @Kevin Kho @Zanie @George Coyne I redeployed the helm with a different name as "*prefect_serverr",*previously it was prefect_server
I was succefull in deploting and in kubernetics engine I can see those pods are running.But the UI is still not active same error in UI,How can I resolve this
k

Kevin Kho

02/14/2022, 3:08 PM
Hi @Sankalp Patnaik, no need to tag multiple team members like that. It’s hard to say what is going on here. Do you have some additional setup beyond the standard helm chart?
s

Sankalp Patnaik

02/14/2022, 3:33 PM
Yes Kevin we are using Docker to buils an image where I have added agent local and prefect server start,apart from this Kubernetics to spin the pod
k

Kevin Kho

02/14/2022, 4:09 PM
There are some things that look off here. 1. You shoudn’t need to call
prefect server start
if you are already spinning up server using helm.
prefect server start
spins up the services as Docker containers with
docker-compose
while using the helm spins up the services and Kubernetes deployments. They are two ways to deploy and you don’t need both 2. I don’t think you should be calling
prefect server start
inside a Dockerfile. Again,
prefect server start
uses Docker compose so this will run docker in docker. It’s not intended to be run inside a container.
s

Sankalp Patnaik

02/14/2022, 4:20 PM
I have one doubt Kevin in below image Prefect-server helm was created last year which stopped working Prefect-serverr helm(2nd one) highlighted created by me last week which was able to spin pods in GKE Should I create the helm chart with Prefect-server only or any name should work.I fear I am using an additional r(highlighted) at athe end which is causing problems
k

Kevin Kho

02/14/2022, 4:28 PM
Per the readme under number 4, I think it should be fine. The previous installation you had of server seems to have been against the master branch like the original issue with Dave here. The setup you are doing should be more stable
s

Sankalp Patnaik

02/23/2022, 10:19 AM
Hi @Kevin Kho I used 
prefect server config
 to generate the Docker Compose file and then 
docker compose up
 to spin up the server's services. When I tried to start the Agent, I got the following errorprefect.utilities.exceptions.ClientError [{'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'locations': [{'line': 2, 'column': 5}], 'path': ['project'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'type': 'system', 'errno': 'ECONNREFUSED', 'code': 'ECONNREFUSED'}}}]
am i missing something while spinning in docker
4 Views