Thread
#prefect-community
    Dave

    Dave

    7 months ago
    Hi team, our team has recently deployed prefect on gcp kubernete. and it keeps saying
    Couldn't connect to *Prefect Server* at *<http://34.82.119.23:4200/graphql>
    ,* can anyone help?
    Kevin Kho

    Kevin Kho

    7 months ago
    Is this message when you go to the UI?
    Dave

    Dave

    7 months ago
    yeah
    this error only occurs after recent prefect UI change
    Kevin Kho

    Kevin Kho

    7 months ago
    There doesn’t seem to be any change that would be related. What was your previous version and current version?
    Dave

    Dave

    7 months ago
    UI Release Jan 14 • 6:50AM
    Kevin Kho

    Kevin Kho

    7 months ago
    What was the previous? Is your API pod healthy?
    Dave

    Dave

    7 months ago
    all status seems good
    the same flow was running well until yesterday
    Kevin Kho

    Kevin Kho

    7 months ago
    Am a bit confused. I thought the error was from the UI but your UI is loading?
    Dave

    Dave

    7 months ago
    Kevin Kho

    Kevin Kho

    7 months ago
    Ah I see
    Are your agents still running? Able to hit the API?
    Dave

    Dave

    7 months ago
    what agents? and how to hit the api?
    sry Im kinda new
    Kevin Kho

    Kevin Kho

    7 months ago
    The agents are the things that run the Flows so you must have had some online at some point. Actually ignore that. Could you try going to the GraphQL playground at http://34.82.119.23:4200?
    Dave

    Dave

    7 months ago
    yes I can
    Kevin Kho

    Kevin Kho

    7 months ago
    Could you try the hello world query?
    query {
        hello
    }
    Dave

    Dave

    7 months ago
    only got error msg
    it says
    connect ECONNREFUSED 10.132.14.47:4201
    Kevin Kho

    Kevin Kho

    7 months ago
    There looks to be something down in the backend. I don’t think this is a UI issue. Could you check the Hasura and GraphQL pod for logs? And I guess might as well check the database is good? These are all in the made k8s cluster right?
    Just asking on the cluster because someone had a CORS issue recently but they were doing cross-cluster things.
    Dave

    Dave

    7 months ago
    Could not upgrade the database!
    Kevin Kho

    Kevin Kho

    7 months ago
    Maybe try running the database migration manually from the command here ?
    Dave

    Dave

    7 months ago
    where should I run these command?
    Michael Adkins

    Michael Adkins

    7 months ago
    It looks like your GraphQL service isn’t starting because the init container is failing to run migrations. Are there logs in your hasura pod?
    Kevin Kho

    Kevin Kho

    7 months ago
    One sec let me ask someone. The answer is the graphql pod but it might not help you
    Dave

    Dave

    7 months ago
    Sure, take your time!
    Michael Adkins

    Michael Adkins

    7 months ago
    I see some info level logs without any helpful errors in that screenshot, hm.
    What helm chart version are you on? Hasura was recently upgraded.
    Dave

    Dave

    7 months ago
    where can i check helm chart version?
    Michael Adkins

    Michael Adkins

    7 months ago
    helm list
    should do it
    Dave

    Dave

    7 months ago
    hmm, I'm on GCP and where should I got to run the command?
    Michael Adkins

    Michael Adkins

    7 months ago
    Ah from a terminal, you can use the gcloud cli to configure kubectl to connect correctly
    There also may be versioning information in the pod labels or something
    Dave

    Dave

    7 months ago
    prefect-server-hasura was created Apr 14, 2021
    apiVersion: v1 kind: Service metadata: annotations: cloud.google.com/neg: '{"ingress":true}' creationTimestamp: "2021-04-14T21:09:52Z" labels: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: prefect-server app.kubernetes.io/version: latest helm.sh/chart: prefect-server-0.0.0 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:app.kubernetes.io/version: {} f:helm.sh/chart: {} f😒pec: f😛orts: .: {} k:{"port":3000,"protocol":"TCP"}: .: {} f:name: {} f😛ort: {} f😛rotocol: {} f:targetPort: {} f😒elector: .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/name: {} f😒essionAffinity: {} f:type: {} manager: Go-http-client operation: Update time: "2021-04-14T21:09:52Z" name: prefect-server-hasura namespace: default resourceVersion: "8964515" uid: 47ddfda5-3ac6-49f7-8fd6-c07a818242b1 spec: clusterIP: 10.132.1.243 clusterIPs: - 10.132.1.243 ports: - name: http port: 3000 protocol: TCP targetPort: http selector: app.kubernetes.io/component: hasura app.kubernetes.io/instance: prefect-server app.kubernetes.io/name: prefect-server sessionAffinity: None type: ClusterIP status: loadBalancer: {}
    this is the yaml file, does it help?
    NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prefect-server default 8 2021-04-14 16:07:02.31896 -0700 -0700 deployed prefect-server-0.0.0 latest
    Michael Adkins

    Michael Adkins

    7 months ago
    Ah it looks like you’ve installed from GitHub instead of the released helm chart
    Dave

    Dave

    7 months ago
    the guy who installed it has left the team 😂
    Michael Adkins

    Michael Adkins

    7 months ago
    I see haha, it looks like a pretty old version but that shouldn’t matter
    What is the image tag for the hasura pod?
    It seems likely that since you’re on the “latest” release instead of a version pin your GraphQL container has been updated to the newest version which is compatible with Hasura 2.x but your Hasura container is still running 1.x
    Dave

    Dave

    7 months ago
    so should I update Hasura?
    im wondering why it happends these two days
    actually there was no issue before that
    Michael Adkins

    Michael Adkins

    7 months ago
    If you pod rolled over it could grab the latest image
    I’d recommend installing the helm chart from a release 🙂
    Dave

    Dave

    7 months ago
    will it impact anything?
    and how to install?
    Michael Adkins

    Michael Adkins

    7 months ago
    I’m not sure how to do the upgrade without affecting your current services, it depends on how your chart was deployed in the first place.
    Basically: You should be fine if you ensure the same database is used, but I’m not sure how your database was configured and don’t want to try to give advice from over here.
    Whoever installed your chart did so from the development version, which is unfortunate.
    Dave

    Dave

    7 months ago
    do you think the problem can be solved by restalling the helm chart?
    Kevin Kho

    Kevin Kho

    7 months ago
    Hopefully cuz your versions across the services would now be compatible.
    Dave

    Dave

    7 months ago
    Thank you, Michael and Kevin, for helping! I will let you know if have any issue while installing.
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    How can I find the config.yaml or value.yaml file location if I was not present at the time of installation
    Kevin Kho

    Kevin Kho

    7 months ago
    Do you mean
    config.toml
    ? What are you trying to do?
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    yes Kevin,I was trying to upgrade helm chart but it failed with "Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: RoleBinding "prefect-server-agent-role-binding" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "prefect-server"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"" I guess we may need to check the toml file
    The error we are getting in GKE workload
    Kevin Kho

    Kevin Kho

    7 months ago
    Ah my bad I see what you are saying. The values.yaml comes from here, but I don’t know hot to get the
    config.yaml
    that was used. Because it’s something you have to continue to provide
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    I was trying to get more information so i checked the logs of pod and got attached error.So now what can be the resolution ?
    Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml Could not upgrade the database! Error: Failed to apply Hasura metadata! Error: 400 Client Error: Bad Request for url: http://prefect-server-hasura.default:3000/v1/query
    Kevin Kho

    Kevin Kho

    7 months ago
    I’d have to ask Michael when he’s around
    Dave

    Dave

    7 months ago
    Hi Kevin, is Michael around?
    Michael Adkins

    Michael Adkins

    7 months ago
    Hey Dave, is this after upgrading the chart to a pinned version?
    You can also try updating the tags on the containers from “latest” to an older version
    Anything before we upgraded Hasura should work
    Dave

    Dave

    7 months ago
    Hey Michael, actually I meet such issue when update helm
    Michael Adkins

    Michael Adkins

    7 months ago
    Looks like you didn’t run
    helm repo update
    Dave

    Dave

    7 months ago
    lemme try it
    Michael Adkins

    Michael Adkins

    7 months ago
    You’re probably better off pinning those containers to an old version rather than upgrading though
    If you don’t know how it’s setup, you could delete your database.
    Dave

    Dave

    7 months ago
    how to pin those containers to an older version?
    Michael Adkins

    Michael Adkins

    7 months ago
    You’ll need to change the tag specified in the deployment from “latest” to a version
    Dave

    Dave

    7 months ago
    can i directly change it in yaml file?
    Michael Adkins

    Michael Adkins

    7 months ago
    Yeah if GCP supports that! But you want to change the image tag not the label.
    Dave

    Dave

    7 months ago
    sorry I'm still confused, can we have a video call if you dont mind?
    Michael Adkins

    Michael Adkins

    7 months ago
    I cannot. You’ll have to go through @George Coyne
    Dave

    Dave

    7 months ago
    alright, no problem
    how can I change the image tag?
    Kevin Kho

    Kevin Kho

    7 months ago
    unless Michael corrects me, I think you should see a containers section there with the graphql container and this will have an image. Hasura was upgraded to 2.0 here and this likely broke the setup so you need to pin all of the Prefect containers to something like over a month ago. Maybe the
    core-0.15.11
    tag for example for the server image. You then need to do this for all the Prefect containers. For graphql for example, you would replace this line
    George Coyne

    George Coyne

    7 months ago
    We just hopped off of a webex, we think we found the values.yml that was used to deploy the current instance and he knows where to make tag changes.
    Dave

    Dave

    7 months ago
    Hey, do we need to re-deploy after making the tag changes?
    Kevin Kho

    Kevin Kho

    7 months ago
    I would think so
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    Hi @Kevin Kho @Michael Adkins @George Coyne I redeployed the helm with a different name as "*prefect_serverr",*previously it was prefect_server
    I was succefull in deploting and in kubernetics engine I can see those pods are running.But the UI is still not active same error in UI,How can I resolve this
    Kevin Kho

    Kevin Kho

    7 months ago
    Hi @Sankalp Patnaik, no need to tag multiple team members like that. It’s hard to say what is going on here. Do you have some additional setup beyond the standard helm chart?
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    Yes Kevin we are using Docker to buils an image where I have added agent local and prefect server start,apart from this Kubernetics to spin the pod
    Kevin Kho

    Kevin Kho

    7 months ago
    There are some things that look off here.1. You shoudn’t need to call
    prefect server start
    if you are already spinning up server using helm.
    prefect server start
    spins up the services as Docker containers with
    docker-compose
    while using the helm spins up the services and Kubernetes deployments. They are two ways to deploy and you don’t need both 2. I don’t think you should be calling
    prefect server start
    inside a Dockerfile. Again,
    prefect server start
    uses Docker compose so this will run docker in docker. It’s not intended to be run inside a container.
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    I have one doubt Kevin in below image Prefect-server helm was created last year which stopped working Prefect-serverr helm(2nd one) highlighted created by me last week which was able to spin pods in GKE Should I create the helm chart with Prefect-server only or any name should work.I fear I am using an additional r(highlighted) at athe end which is causing problems
    Kevin Kho

    Kevin Kho

    7 months ago
    Per the readme under number 4, I think it should be fine. The previous installation you had of server seems to have been against the master branch like the original issue with Dave here. The setup you are doing should be more stable
    Sankalp Patnaik

    Sankalp Patnaik

    7 months ago
    Hi @Kevin Kho I used 
    prefect server config
     to generate the Docker Compose file and then 
    docker compose up
     to spin up the server's services. When I tried to start the Agent, I got the following error😛refect.utilities.exceptions.ClientError: [{'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'locations': [{'line': 2, 'column': 5}], 'path': ['project'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to http://prefect-server-hasura.default:3000/v1alpha1/graphql failed, reason: connect ECONNREFUSED 10.132.1.243:3000', 'type': 'system', 'errno': 'ECONNREFUSED', 'code': 'ECONNREFUSED'}}}]
    am i missing something while spinning in docker