https://prefect.io
Join Slack
Hi team, our team has recently deployed prefect on gcp kubernete. and it keeps saying `Couldn't conn...
# ask-community
d
Hi team, our team has recently deployed prefect on gcp kubernete. and it keeps saying 
Couldn't connect to *Prefect Server* at *<http://34.82.119.23:4200/graphql>
,* can anyone help?
k
Is this message when you go to the UI?
d
yeah
this error only occurs after recent prefect UI change
k
There doesn’t seem to be any change that would be related. What was your previous version and current version?
d
UI Release Jan 14 • 6:50AM
k
What was the previous? Is your API pod healthy?
d
all status seems good
the same flow was running well until yesterday
k
Am a bit confused. I thought the error was from the UI but your UI is loading?
d
k
Ah I see
Are your agents still running? Able to hit the API?
d
what agents? and how to hit the api?
sry Im kinda new
k
The agents are the things that run the Flows so you must have had some online at some point. Actually ignore that. Could you try going to the GraphQL playground at http://34.82.119.23:4200?
d
yes I can
k
Could you try the hello world query?
Copy code
query {
    hello
}
d
only got error msg
it says 
connect ECONNREFUSED 10.132.14.47:4201
k
There looks to be something down in the backend. I don’t think this is a UI issue. Could you check the Hasura and GraphQL pod for logs? And I guess might as well check the database is good? These are all in the made k8s cluster right?
Just asking on the cluster because someone had a CORS issue recently but they were doing cross-cluster things.
d
Could not upgrade the database!
k
Maybe try running the database migration manually from the command here ?
d
where should I run these command?
z
It looks like your GraphQL service isn’t starting because the init container is failing to run migrations. Are there logs in your hasura pod?
k
One sec let me ask someone. The answer is the graphql pod but it might not help you
d
Sure, take your time!
z
I see some info level logs without any helpful errors in that screenshot, hm.
What helm chart version are you on? Hasura was recently upgraded.
d
where can i check helm chart version?
z
helm list
should do it
d
hmm, I'm on GCP and where should I got to run the command?
z
Ah from a terminal, you can use the gcloud cli to configure kubectl to connect correctly
There also may be versioning information in the pod labels or something
d
prefect-server-hasura was created Apr 14, 2021
apiVersion: v1 kind: Service metadata: annotations: cloud.google.com/neg: '{"ingress":true}' creationTimestamp: "2021-04-14T210952Z" labels: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: prefect-server app.kubernetes.io/version: latest helm.sh/chart: prefect-server-0.0.0 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: fmetadata flabels .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:app.kubernetes.io/version: {} f:helm.sh/chart: {} fspec fports .: {} k:{"port":3000,"protocol":"TCP"}: .: {} fname {} fport {} fprotocol {} ftargetPort {} fselector .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/name: {} fsessionAffinity {} ftype {} manager: Go-http-client operation: Update time: "2021-04-14T210952Z" name: prefect-server-hasura namespace: default resourceVersion: "8964515" uid: 47ddfda5-3ac6-49f7-8fd6-c07a818242b1 spec: clusterIP: 10.132.1.243 clusterIPs: - 10.132.1.243 ports: - name: http port: 3000 protocol: TCP targetPort: http selector: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/name: prefect-server sessionAffinity: None type: ClusterIP status: loadBalancer: {}
this is the yaml file, does it help?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prefect-server default 8 2021-04-14 160702.31896 -0700 -0700 deployed prefect-server-0.0.0 latest
z
Ah it looks like you’ve installed from GitHub instead of the released helm chart
d
the guy who installed it has left the team 😂
z
I see haha, it looks like a pretty old version but that shouldn’t matter
What is the image tag for the hasura pod?
It seems likely that since you’re on the “latest” release instead of a version pin your GraphQL container has been updated to the newest version which is compatible with Hasura 2.x but your Hasura container is still running 1.x
d
so should I update Hasura?
im wondering why it happends these two days
actually there was no issue before that
z
If you pod rolled over it could grab the latest image
I’d recommend installing the helm chart from a release 🙂
d
will it impact anything?
and how to install?
z
I’m not sure how to do the upgrade without affecting your current services, it depends on how your chart was deployed in the first place.
Basically: You should be fine if you ensure the same database is used, but I’m not sure how your database was configured and don’t want to try to give advice from over here.
Whoever installed your chart did so from the development version, which is unfortunate.
d
do you think the problem can be solved by restalling the helm chart?
k
Hopefully cuz your versions across the services would now be compatible.
d
Thank you, Michael and Kevin, for helping! I will let you know if have any issue while installing.
s
How can I find the config.yaml or value.yaml file location if I was not present at the time of installation
k
Do you mean 
config.toml
? What are you trying to do?
s
yes Kevin,I was trying to upgrade helm chart but it failed with "Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: RoleBinding "prefect-server-agent-role-binding" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "prefect-server"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"" I guess we may need to check the toml file
The error we are getting in GKE workload
k
Ah my bad I see what you are saying. The values.yaml comes from here, but I don’t know hot to get the 
config.yaml
that was used. Because it’s something you have to continue to provide
s
I was trying to get more information so i checked the logs of pod and got attached error.So now what can be the resolution ?
Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml Could not upgrade the database! Error: Failed to apply Hasura metadata! Error: 400 Client Error: Bad Request for url: http://prefect-server-hasura.default:3000/v1/query
k
I’d have to ask Michael when he’s around
d
Hi Kevin, is Michael around?
z
Hey Dave, is this after upgrading the chart to a pinned version?
You can also try updating the tags on the containers from “latest” to an older version
Anything before we upgraded Hasura should work
d
Hey Michael, actually I meet such issue when update helm
z
Looks like you didn’t run 
helm repo update
d
lemme try it
z
You’re probably better off pinning those containers to an old version rather than upgrading though
If you don’t know how it’s setup, you could delete your database.
d
how to pin those containers to an older version?
z
You’ll need to change the tag specified in the deployment from “latest” to a version
d
can i directly change it in yaml file?
z
Yeah if GCP supports that! But you want to change the image tag not the label.
d
sorry I'm still confused, can we have a video call if you dont mind?
z
I cannot. You’ll have to go through @George Coyne
d
alright, no problem
how can I change the image tag?
k
unless Michael corrects me, I think you should see a containers section there with the graphql container and this will have an image. Hasura was upgraded to 2.0 here and this likely broke the setup so you need to pin all of the Prefect containers to something like over a month ago. Maybe the 
core-0.15.11
tag for example for the server image. You then need to do this for all the Prefect containers. For graphql for example, you would replace this line
g
We just hopped off of a webex, we think we found the values.yml that was used to deploy the current instance and he knows where to make tag changes.
d
Hey, do we need to re-deploy after making the tag changes?
k
I would think so
s
Hi @Kevin Kho @Zanie @George Coyne I redeployed the helm with a different name as "*prefect_serverr",*previously it was prefect_server
I was succefull in deploting and in kubernetics engine I can see those pods are running.But the UI is still not active same error in UI,How can I resolve this
k
Hi @Sankalp Patnaik, no need to tag multiple team members like that. It’s hard to say what is going on here. Do you have some additional setup beyond the standard helm chart?
s
Yes Kevin we are using Docker to buils an image where I have added agent local and prefect server start,apart from this Kubernetics to spin the pod
k
There are some things that look off here. 1. You shoudn’t need to call 
prefect server start
if you are already spinning up server using helm. 
prefect server start
spins up the services as Docker containers with 
docker-compose
while using the helm spins up the services and Kubernetes deployments. They are two ways to deploy and you don’t need both 2. I don’t think you should be calling 
prefect server start
inside a Dockerfile. Again, 
prefect server start
uses Docker compose so this will run docker in docker. It’s not intended to be run inside a container.
s
I have one doubt Kevin in below image Prefect-server helm was created last year which stopped working Prefect-serverr helm(2nd one) highlighted created by me last week which was able to spin pods in GKE Should I create the helm chart with Prefect-server only or any name should work.I fear I am using an additional r(highlighted) at athe end which is causing problems
k
Per the readme under number 4, I think it should be fine. The previous installation you had of server seems to have been against the master branch like the original issue with Dave here. The setup you are doing should be more stable
s
Hi @Kevin Kho I used 
prefect server config
 to generate the Docker Compose file and then 
docker compose up
 to spin up the server's services. When I tried to start the Agent, I got the following errorprefect.utilities.exceptions.ClientError [{'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'locations': [{'line': 2, 'column': 5}], 'path': ['project'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'type': 'system', 'errno': 'ECONNREFUSED', 'code': 'ECONNREFUSED'}}}]
am i missing something while spinning in docker
12 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by