https://prefect.io logo
r

Riley Hun

11/25/2020, 6:10 PM
Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.
Here's my config:
Copy code
postgresql:
  postgresqlDatabase: prefect
  postgresqlUsername: postgres
  postgresqlPassword: <PASSWORD>
  servicePort: 5432
  externalHostname: <IP_ADDRESS>
  useSubChart: false
The error I'm getting:
Copy code
secret "prefect-gke-3-postgresql" not found: CreateContainerConfigError
z

Zanie

11/25/2020, 6:25 PM
I’ll try to reproduce this, but it looks like it is failing to generating the secret to place the password in
It’d be best to put the password in your cluster as a secret then refer to it using
existingSecret
anyway, could you give that a try?
r

Riley Hun

11/25/2020, 6:53 PM
Sure @Zanie - I'll give that a try
z

Zanie

11/25/2020, 7:24 PM
Yeah so it looks like
existingSecret: null
only generates a secret for you if
useSubChart: true
From the values.yaml docs:
Copy code
# existingSecret configures which secret should be referenced
  # for access to the database. If null and `useSubChart` is
  # enabled, the secret will be generated. If using an external
  # postgres service, this value should be set to a secret
  # containing the key `postgresql-password`
I could update this behavior to create the secret for you if you’re not using the subchart still
r

Riley Hun

11/25/2020, 8:20 PM
Thanks for this information @Zanie. Yeah I'm not using the subchart - I'm using an external db. But I can also try by setting a secret to the kubernetes cluster and setting existingSecret in the config.
@Zanie My pods are having some trouble initializing. I did this: Setting existing password to cluster
Copy code
kubectl create secret generic prefect-db-secrets   --from-literal=postgresql-password=${DB_PASSWORD}
Config
Copy code
postgresqlDatabase: prefect
  postgresqlUsername: postgres
  existingSecret: prefect-db-secrets
  servicePort: 5432
  externalHostname: 34.74.17.161
  useSubChart: false
Note: I am using a private cloudsql instance, but my cluster is in the same VPC network.
z

Zanie

11/25/2020, 8:40 PM
When you get inspect the pods with
kubectl describe pod <pod>
or
kubectl logs <pod>
what are the errors?
r

Riley Hun

11/25/2020, 8:41 PM
Was just about to send you that: From pod kubectl logs prefect-gke-3-hasura-66ccdd566-qr6mg
Copy code
{"type":"startup","timestamp":"2020-11-25T20:39:10.209+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}}
{"type":"startup","timestamp":"2020-11-25T20:39:10.209+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"<postgresql://postgres@34.74.17.161:5432/prefect>"}}}
{"type":"pg-client","timestamp":"2020-11-25T20:39:42.035+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
I could try again by making the cloudsql instance public?
z

Zanie

11/25/2020, 8:43 PM
What are the logs in the postgres pod?
r

Riley Hun

11/25/2020, 8:55 PM
I don't see a postgres pod anymore after switching to using an external db
z

Zanie

11/25/2020, 9:00 PM
Oh duh haha sorry. Can you look at the logs for the managed postgres and see if it is denying the connection?
Might also be worth trying to exec into one of the pods in your cluster and seeing if you can access the hostname (https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/)
r

Riley Hun

11/25/2020, 9:33 PM
@Zanie Side question - do we need to set a service account to the GKE cluster with cloudsql client permissions?
z

Zanie

11/25/2020, 9:43 PM
So I’ve looked through the GKE CloudSQL docs and it looks like there are two options • Using a proxy — which requires a service account that you register with an associated Google Service Account for permissioning and a container that handles the proxy • Using VPC-native GKE cluster and connecting to a Cloud SQL instance on the same VPC using private IP without the proxy
The helm chart does provide a service account or the ability to refer to an existing one, but you’d have to attach the relevant permissions to it and then set up the container for the proxy yourself.
With the private IP, it should work when set as the
externalHostname
I will open an issue in the server repo to add more robust support or instructions for setting up an external postgres per cloud provider, but since each provider has their own requirements this is not going to be added to the helm chart quickly.
r

Riley Hun

11/25/2020, 10:27 PM
Ok that would be great @Zanie. In terms of next immediate steps on my side, what would you recommend?
Also, pull up the postgres logs for my cloudsql instance, and no errors identified.
z

Zanie

11/25/2020, 10:31 PM
I would recommend trying the private ip method. With the builtin postgres, it usually will log failed connection attempts so I presume your issue is that the hasura pod cannot see the cloudsql server at the ip you provided
r

Riley Hun

11/25/2020, 10:38 PM
Oh! I think that's the method I'm trying to do right now. I'm already using a private ip and setting it to the
external hostname
in the helm chart config.
Ohhh oopsies - I was using the wrong IP address for my SQL instance. 🤦🤦🤦
Yaaay! It worked. Thanks @Zanie. The issue was I used the wrong private IP address. Very careless of me.
z

Zanie

11/25/2020, 11:11 PM
Glad it’s working! 😄
r

Riley Hun

11/25/2020, 11:14 PM
If you have time, could I also ask another semi-related question? This is the last thing I'm trying to figure out with respect to the helm chart.
It's interesting to me that I'm able to connect to a private cloudsql instance without any issue. But, I'm trying to do something similar with the apollo endpoint. That is, I made the graphql-hasura service an internal load balancer on the same VPC network. But unfortunately, the UI is having trouble connecting to it.
z

Zanie

11/25/2020, 11:18 PM
Is that the question?
r

Riley Hun

11/25/2020, 11:19 PM
Accidentally hit enter too fast 🤣
z

Zanie

11/25/2020, 11:20 PM
So the UI needs to be accessible from your (the user’s) browser because it’s client side not server side code
The default value is set in the config,
Copy code
# apolloApiUrl defines the default gateway to the Apollo
  # GraphQL server. This location must be accessible by the
  # user of the API because the browser is making requests
  # (not the ui server)
  apolloApiUrl: <http://localhost:4200/graphql>
r

Riley Hun

11/25/2020, 11:21 PM
So what is the best way to protect the apollo url because right now it's exposed to the internet. I tried both nginx-ingress and internal load balancer and neither of those approaches worked. I also know that hasura-graphql has its own authentication set through an environmental variable, but pretty sure need to make additional changes in the code for this to work
z

Zanie

11/25/2020, 11:23 PM
That’s kind of an open question because auth is really complicated / tied to specific infrastructure. This is part of the reason we made Prefect Cloud so that we can provide a solution to this but it’s not something we can solve for each user of Server.
Currently I’d recommend putting your cluster behind a VPN or using
kubectl port-forward
to make the apollo api available at
localhost:4200
Once the helm chart is a bit more mature we’ll likely have clearer documentation / examples for this part
r

Riley Hun

11/25/2020, 11:27 PM
It's already on a private cluster connected to a VPC network. I had hoped that since the apollo service is an internal load balancer that's on the same VPC as the cluster, there would be no issues connecting to it. Or, is this flawed thinking?
As for your recommendation, I could try that as a potential next step
z

Zanie

11/25/2020, 11:32 PM
It looks to me like you have two questions here but correct me if I’m wrong • How do I protect the apollo service from external acccess? — if it’s hosted on a private VPC you’ve already accomplished this. If not, may want to set up a VPN that is whitelisted to access your VPC and blacklist all other access. • How do I get the UI to communicate with the apollo service api? — the apollo service must be accessible to the user that is viewing the UI. I would recommend either using
kubectl port-forward
or if the user can see the external ip of the service changing the graphql target url in the UI home screen
r

Riley Hun

11/25/2020, 11:38 PM
Do I need to make adjustments in the helm chart configuration? Example:
Copy code
ui:
  image:
    name: prefecthq/ui
    tag: null
    pullPolicy: IfNotPresent
    pullSecrets: []
  apolloApiUrl: http://<Internal_Load_Balancer_IP>:4200/graphql
z

Zanie

11/25/2020, 11:39 PM
So the
apolloApiUrl
is setting the default for each user of the UI but they can override it from the UI home screen
The IP would have to be the external name though, not the internal
r

Riley Hun

11/25/2020, 11:45 PM
Ok first of all, thanks for all these helpful directions. I guess let me make note of all this information and digest it. Networking is not my specialty. So I'll need to do some research for more info on a lot of what you said above. But my preliminary thoughts are that this doesn't seem very seamless if the user has to port forward the internal address.
z

Zanie

11/25/2020, 11:48 PM
If the apollo api is externally available but limited to some whitelist of IPs then you could set the default
apolloApiUrl
default and the user wouldn’t have to do anything.
r

Riley Hun

11/25/2020, 11:49 PM
So basically, the answer is create a firewall for the external load balancer of the apollo api and limit it to specific IP addresses?
z

Zanie

11/25/2020, 11:49 PM
I think that’d be the easiest solution
r

Riley Hun

11/25/2020, 11:51 PM
Okay I will do that then. Makes a lot of sense. Thanks for the suggestion For the UI, I will create an ingress with basic auth. The UI seems to be independent of the other components so nothing should break if I add that security layer to the UI.
3 Views