https://prefect.io logo
#prefect-ui
Title
# prefect-ui
r

Rebecca Servaites

05/10/2021, 8:17 PM
Hello everyone!! I am having some issues getting the UI to run correctly on a Kubernetes cluster, deployed with the Prefect Helm chart I am using the L7 ingress for apollo and the UI and I keep getting this error: I am able to register projects and flows with the CLI, so I think everything is connecting except for the UI. Here's my Helm chart: Any ideas on where I might be going wrong?
k

Kevin Kho

05/10/2021, 8:19 PM
Hi @Rebecca Servaites! I’ll forward this to the team. Is it possible to move the config to the thread to save space on the main channel?
r

Rebecca Servaites

05/10/2021, 8:20 PM
Hi @Kevin Kho Sure!! I'm not a regular Slack user, so you'll have to tell me how to do that. :)
I figured it out!! Lol 😉
Copy code
# prefect-server configuration

# serverVersionTag configures the default tag for prefect server
# images which are used for the apollo, graphql, and towel
# services. If null, this value defaults to `appVersion` but is
# provided here to allow override at chart install time. Each
# service image tag can be overriden individually in their 
# configurations but it is not recommended.
# See <https://github.com/helm/helm/issues/8194> for details
# on why the `appVersion` cannot be set at install time.
serverVersionTag: null

# prefectVersionTag configures the default tag for Prefect Core based
# services, including the agent and default tenant creation job.
# This value is automatically pinned on chart release to the latest
# core version.
prefectVersionTag: "latest"

# uiVersionTag configures the default tag for the Prefect UI service.
# It is defined here for easy update using `sed` for automation.
# This value is automatically pinned on chart release to the latest
# ui version.
uiVersionTag: "latest"

# imagePullSecrets provides configuration to reference the k8s Secret
# resources the Helm chart's pods can get credentials from to pull
# their images.
imagePullSecrets: []

# annotations to merge into all object configurations
# NOTE: These will not apply to the postgresql subchart and must be
# defined within `postgresql` as well if that is desired
annotations: {}

# postgresql configures the backing database
# NOTE: the variables names must remain as is to match the
# bitnami subchart even if the bitnami subchart is not being
# used, these variables are used to create the connection
# string
postgresql:
  postgresqlDatabase: prefect

  # postgresqlUsername defines the username to authenticate
  # with.
  # NOTE: If you are using Azure, this will include an '@'
  # which must be encoded as '%40' for the connection string
  # to work with both Hasura, the GraphQL server, and the
  # Alembic migration manager
  postgresqlUsername: prefect

  # existingSecret configures which secret should be referenced
  # for access to the database. If null and `useSubChart` is
  # enabled, the secret will be generated. If using an external
  # postgres service, this value should be set to a secret
  # containing the key `postgresql-password`
  existingSecret: null

  # postgresqlPassword sets the password to be used if
  # `existingSecret` is not set. This is the password for
  # `postgresqlUsername` and will be set within the secret at
  # the key `postgresql-password`
  postgresqlPassword: my-super-secret-password

  # servicePort configures the port that the database should be
  # accessed at
  servicePort: 5432

  # externalHostname defines the address to contact an externally
  # managed postgres database instance at. This is not required if
  # `internalPostgres` is `true`
  externalHostname: ""

  # useSubChart determines if a this chart should deploy a
  # user-manager postgres database or use an externally managed
  # postgres instance. If `useSubChart` is `true`, the
  # bitnami/postgresql subchart will be deployed
  useSubChart: true

  # postgresql configuration below here is only used if using
  # the subchart

  # persistence enables a PVC that stores the database between
  # deployments. If making changes to the database deployment, this
  # PVC will need to be deleted for database changes to take effect.
  # This is especially notable when the authentication password
  # changes on redeploys.
  # This is disabled by default because we do not recommend using
  # the subchart deployment for production deployments.
  persistence:
    enabled: false
    size: 8Gi

  initdbUser: postgres

  initdbScripts:
    create_pgcrypto.sql: |
      -- create pgcrypto extension, required for Hasura UUID
      CREATE EXTENSION IF NOT EXISTS pgcrypto;
      CREATE EXTENSION IF NOT EXISTS "pg_trgm";
      SET TIME ZONE 'UTC';

# prefectConfig allows override of prefect_server/config.toml values
# Values placed here will result in environment variable definitions
# starting with "PREFECT_SERVER__" and will be passed to relevant
# services
prefectConfig:
  # env: "local"
  # debug: "false"
  # queued_runs_returned_limit: "25"
  # hasura:
  #   execute_retry_seconds: "10"
  # logging:
  #   level: "DEBUG"
  #   format: "[%(asctime)s] %(levelname)s - %(name)s | %(message)s"
  # services:
  #   scheduler:
  #     scheduler_loop_seconds: "300"
  #   lazarus:
  #     resurrection_attempt_limit: "3"
  # telemetry:
  #   enabled: "true"


# hasura configures the Prefect hasura deployment and service
# which creates a graphql api from the postgres database
hasura:

  # hasura.image configures the docker image used for hasura
  # and is the only image in the chart that is not hosted by
  # Prefect
  image:
    name: hasura/graphql-engine
    tag: v1.3.3
    pullPolicy: IfNotPresent
    pullSecrets: []

  service:
    # type defines the service type and defaults to ClusterIP
    # because this service does not need to be exposed outside
    # the cluster
    type: ClusterIP
    port: 3000

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}


# graphql configures the Prefect graphql deployment and service
# which provides a python graphql server on starlette
graphql:

  image:
    name: prefecthq/server
    tag: null
    pullPolicy: Always
    pullSecrets: []

  service:
    type: ClusterIP
    port: 4201

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}


  init:
    # init.resources configures resources for the initContainer
    # which upgrades the database
    resources: {}

# apollo configures the Prefect apollo deployment and service
# which provides a unified graphql schema for users and the UI
# to interact with
apollo:
  image:
    name: prefecthq/apollo
    tag: null
    pullPolicy: Always
    pullSecrets: []
  options:
    telemetryEnabled: true

  service:
    # type defines the service type and defaults to LoadBalancer
    # to expose the apollo service to users and the UI.
    # type: LoadBalancer
    type: LoadBalancer
    port: 4200

  ingress:
    enabled: true

    # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
    # See <https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress>
    # ingressClassName: nginx

    annotations: {}
    labels: {}

    ## Hosts must be provided if Ingress is enabled.
    hosts: 
      # - <http://prefecthq-apollo.domain.com|prefecthq-apollo.domain.com>
      - <http://prefect-apollo.mycluster.com|prefect-apollo.mycluster.com>

    ## Path to use for ingress rules
    path: /

    ## TLS configuration for Prefect Ingress
    ## Secret must be manually created in the namespace
    tls: 
    # - secretName: prefecthq-apollo-general-tls
    #   hosts:
    #   - <http://prefecthq-apollo.example.com|prefecthq-apollo.example.com>
    hosts:
      - <http://prefect-apollo.mycluster.com|prefect-apollo.mycluster.com>

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}

# ui configures the Prefect ui deployment and service
ui:

  image:
    name: prefecthq/ui
    tag: null  # See `uiVersionTag` instead
    pullPolicy: Always
    pullSecrets: []

  # apolloApiUrl defines the default gateway to the Apollo
  # GraphQL server. This location must be accessible by the
  # user of the API because the browser is making requests
  # (not the ui server)
  apolloApiUrl: <http://prefect-apollo.mycluster.com>

  service:
    # type defines the service type and defaults to LoadBalancer
    # to expose the ui service to users and the UI.
    # type: LoadBalancer
    type: LoadBalancer
    port: 8080

  ingress:
    enabled: true

    # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
    # See <https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress>
    # ingressClassName: nginx

    annotations: {}
    labels: {}

    ## Hosts must be provided if Ingress is enabled.
    hosts: 
      # - <http://prefecthq-ui.domain.com|prefecthq-ui.domain.com>
      - <http://prefect-ui.mycluster.com|prefect-ui.mycluster.com>

    ## Path to use for ingress rules
    path: /

    ## TLS configuration for Prefect Ingress
    ## Secret must be manually created in the namespace
    tls: 
    # - secretName: prefecthq-ui-general-tls
    #   hosts:
    #   - <http://prefecthq-ui.example.com|prefecthq-ui.example.com>
    hosts:
      - <http://prefect-ui.mycluster.com|prefect-ui.mycluster.com>

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}

# towel configures the Prefect towel deployment which provides
# a group of useful services
towel:
  image:
    name: prefecthq/server
    tag: null
    pullPolicy: Always
    pullSecrets: []

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}


# agent configures an optional Prefect Kubernetes agent which will
# schedule flows as jobs in the cluster
agent:

  # enabled determines if the Prefect Kubernetes agent is deployed
  enabled: false

  # prefectLabels defines what scheduling labels (not K8s labels) should
  # be associated with the agent
  prefectLabels: []

  # image configures the container image for the agent deployment
  image:
    name: prefecthq/prefect
    tag: null
    pullPolicy: Always
    pullSecrets: []

  labels: {}
  annotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}

  # resources defines the computational resources for the Prefect Agent
  resources:
    limits:
      cpu: 100m
      memory: 128Mi

  # job configures values associated with jobs the agent launches
  job:
    # resources defines the computational resources for flow jobs
    resources:
      limits:
        memory: "2Gi"
        cpu: "1Gi"
      requests:
        memory: ""
        cpu: ""

    # imagePullPolicy configures the image pull policy for the flow job
    imagePullPolicy: ""

    # imagePullSecrets defines image pull secrets for the flow job
    # NOTE: These secrets are not merged with the global imagePullSecrets
    imagePullSecrets: []

serviceAccount:
  # create specifies whether a service account should be created to be
  # associated with the prefect-server infrastructure
  create: true

  # name sets the name of the service account to use
  # If not set and create is true, a name is generated using the
  # prefect-server.nameField template
  name: null


# jobs contain one-time job definitions
jobs:

  # create a tenant so that Agent and UI are immediately usable after installation
  createTenant:
    enabled: true

    # tenant sets the details of the created tenant
    tenant:
      name: default
      slug: default

    # image configures the container image for the job
    image:
      name: prefecthq/prefect
      tag: null
      pullPolicy: Always
      pullSecrets: []

    labels: {}
    annotations: {}
    podSecurityContext: {}
    securityContext: {}
    nodeSelector: {}
    tolerations: []
    affinity: {}
    # backoffLimit configures the number of retries; needed to wait for the server to become available
    backoffLimit: 10
k

Kevin Kho

05/10/2021, 8:22 PM
Lol. Thanks!
j

Jenny

05/10/2021, 8:32 PM
HI @Rebecca Servaites - Other users who experienced this have resolved it by using the 2021-04-15 UI image. It's connected to this issue which we are actively working on a fix for: https://github.com/PrefectHQ/ui/issues/798
r

Rebecca Servaites

05/10/2021, 9:22 PM
I'll give that a try and see how it goes! Thanks for the quick reply!! 🙂
👍 1
@Jenny Would you be able to help me some more? I was able to get the UI to spin up, but now it is not connecting to the graphql server. I can access the prefect-apollo.mycluster.com/graphql endpoint from my browser, and it appears to be working. Any idea on what I might be doing wrong? I'm really stuck.
Also, when I try this command from my CLI, to connect to that endpoint I get a 308 code
curl -X POST -H "Content-Type: application/json" -d '{"query": "{ hello }"}' <http://prefect-apollo.mycluster.com/graphql>
j

Jenny

05/11/2021, 8:31 PM
Oh actually - just saw you're using the helm chart - let me check with the team for you too.
r

Rebecca Servaites

05/11/2021, 8:35 PM
Thanks Jenny!! I was just going to say that i configured it in the Helm chart and again in the Getting Started page. It's just not conencting for some reason. 😞
t

Tyler Wanner

05/11/2021, 9:30 PM
hiya what's in that 308 response code? a 308 should be a permanent redirect to somewhere?
r

Rebecca Servaites

05/11/2021, 10:22 PM
It is a Permanent Redirect.
Copy code
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
@Jenny Any ideas from the team on how to fix this one?
j

Jenny

05/13/2021, 2:54 PM
Can you give us anymore information Rebecca? Tyler's question above was trying to get more to go on. Is the HTML that you posted what was sent back as part of the 308? Was there anything else?
t

Tyler Wanner

05/13/2021, 2:57 PM
Sorry Rebecca--I'd like to help more, but I'm tied up at this moment. I am confident we can get this sorted. It seems like a problem with the routing. I haven't been able to try and reproduce yet
r

Rebecca Servaites

05/13/2021, 3:06 PM
The HTML code I posted was the response I got for the curl request, there wan't anything else. The logs haven't been very helpful to me, but I could reproduce and and post those here as well.
t

Tyler Wanner

05/13/2021, 3:45 PM
yea I believe it’s something with nginx in the way of getting to prefect—something about the host or the URI most likely
did you set up any custom DNS for
<http://prefect-apollo.mycluster.com|prefect-apollo.mycluster.com>
r

Rebecca Servaites

05/13/2021, 8:29 PM
@Tyler Wanner My cluster is not public and that is also not the URL to the ingress that I set up - I changed it here for security reasons. I did not setup any custom DNS settings, but I'll have to check with our cluster admin to see if there is anything that could be any strange settings.
t

Tyler Wanner

05/14/2021, 3:35 PM
Hi Rebecca--I sent a few messages but am re-summarizing them here instead... Some notes from my repro 1. for
ui.apolloApiUrl
, I think you'll want to try
<http://prefect-apollo.mycluster.com:4200/graphql|prefect-apollo.mycluster.com:4200/graphql>
2. Please downgrade the UI image by setting
uiVersionTag
to
"core-0.14.19"
--there is a bug in
latest
atm 3. You've defined
ui.ingress.hosts
and
apollo.ingress.hosts
twice--it looks like the second definition should be under
ui.ingress.tls.hosts
and
apollo.ingress.tls.hosts
instead. This won't provide TLS until you add some certificate via a secret.
r

Rebecca Servaites

05/18/2021, 8:38 PM
@Tyler Wanner Thank you for the tips. I haven't gotten it to work yet, so I'll keep at it and see if it connects. In regard to #1, you don't put the port number with the ingress. The service is specified in the ingress template and is forwarded to the defined URL.
@Tyler Wanner I was able to connect tot he server finally, I thought it would connect automatically since I had it defined in the Helm chart, but after I entered the ingress URL in the Prefect UI home page "Prefect Server" it actually connected. Not sure what the difference was, because it looked like that is where it was trying to connect anyway. Thanks for your help! I think the image tag you gave me was what did the trick!
🚀 2
m

Marko Jamedzija

05/21/2021, 10:00 AM
Hello, can anyone explain to me at what time are you releasing
core-X.X.X
versions? On ui Github page I see only
YYYY-MM-DD
releases. Thanks!
plus one 1
r

Rebecca Servaites

05/21/2021, 4:07 PM
I've been wondering trhis as well. I was also using the UI image tagged with a
yyyy-mm-dd
tag.
Well, that connection to the server was a fluke, because I haven't been able to reproduce it.
t

Tyler Wanner

05/21/2021, 9:08 PM
Hi again Rebecca, what do you have set on your ui.apolloApiUrl?
FYI Marko UI images tagged core-X.X.X are published alongside prefecthq/prefect (Prefect Core) releases
👍 1
r

Rebecca Servaites

05/21/2021, 9:36 PM
I actually noticed something interesting with
ui.apolloApiUrl
with the UI tag
core-0.14.19
It only affects the value in the UI Home Page, Prefect Sever Tab Item 2
Prefect Server GraphQL endpoint
text box. When you click the
Reset stored Graphql endpoint
then it shows up. I know this because I wanted to try not having the UI automatically connect to the graphql endpoint on startup and for me to manually enter the URL in the UI home page - so, I set
ui.apolloApiUi
to http://localhost:4200/graphql and the endpoint it kept trying to connect to was the value I entered for
apollo.hosts
which was apollo.mycluster.com with the graphql path. I have all my pods set to
ClusterIP
now because we don't have public IP addresses and the L7 ingress should work for it, because I can access it from my browser. I'm really stumped on why it's not connecting.
16 Views