Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.

I’ll try to reproduce this, but it looks like it is failing to generating the secret to place the password in

Sure @Zanie - I'll give that a try

Yeah so it looks like

existingSecret: null

only generates a secret for you if

useSubChart: true

Thanks for this information @Zanie. Yeah I'm not using the subchart - I'm using an external db. But I can also try by setting a secret to the kubernetes cluster and setting existingSecret in the config.

When you get inspect the pods with

kubectl describe pod <pod>

or

kubectl logs <pod>

what are the errors?

Was just about to send you that: From pod kubectl logs prefect-gke-3-hasura-66ccdd566-qr6mg

{"type":"startup","timestamp":"2020-11-25T20:39:10.209+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}}
{"type":"startup","timestamp":"2020-11-25T20:39:10.209+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"<postgresql://postgres@34.74.17.161:5432/prefect>"}}}
{"type":"pg-client","timestamp":"2020-11-25T20:39:42.035+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}

What are the logs in the postgres pod?

I don't see a postgres pod anymore after switching to using an external db

Oh duh haha sorry. Can you look at the logs for the managed postgres and see if it is denying the connection?

@Zanie Side question - do we need to set a service account to the GKE cluster with cloudsql client permissions?

So I’ve looked through the GKE CloudSQL docs and it looks like there are two options • Using a proxy — which requires a service account that you register with an associated Google Service Account for permissioning and a container that handles the proxy • Using VPC-native GKE cluster and connecting to a Cloud SQL instance on the same VPC using private IP without the proxy

Ok that would be great @Zanie. In terms of next immediate steps on my side, what would you recommend?

I would recommend trying the private ip method. With the builtin postgres, it usually will log failed connection attempts so I presume your issue is that the hasura pod cannot see the cloudsql server at the ip you provided

Oh! I think that's the method I'm trying to do right now. I'm already using a private ip and setting it to the

external hostname

in the helm chart config.

Glad it’s working! 😄

If you have time, could I also ask another semi-related question? This is the last thing I'm trying to figure out with respect to the helm chart.

Is that the question?

Accidentally hit enter too fast 🤣

So the UI needs to be accessible from your (the user’s) browser because it’s client side not server side code

So what is the best way to protect the apollo url because right now it's exposed to the internet. I tried both nginx-ingress and internal load balancer and neither of those approaches worked. I also know that hasura-graphql has its own authentication set through an environmental variable, but pretty sure need to make additional changes in the code for this to work

That’s kind of an open question because auth is really complicated / tied to specific infrastructure. This is part of the reason we made Prefect Cloud so that we can provide a solution to this but it’s not something we can solve for each user of Server.

It's already on a private cluster connected to a VPC network. I had hoped that since the apollo service is an internal load balancer that's on the same VPC as the cluster, there would be no issues connecting to it. Or, is this flawed thinking?

It looks to me like you have two questions here but correct me if I’m wrong • How do I protect the apollo service from external acccess? — if it’s hosted on a private VPC you’ve already accomplished this. If not, may want to set up a VPN that is whitelisted to access your VPC and blacklist all other access. • How do I get the UI to communicate with the apollo service api? — the apollo service must be accessible to the user that is viewing the UI. I would recommend either using

kubectl port-forward

or if the user can see the external ip of the service changing the graphql target url in the UI home screen

Do I need to make adjustments in the helm chart configuration? Example:

ui:
  image:
    name: prefecthq/ui
    tag: null
    pullPolicy: IfNotPresent
    pullSecrets: []
  apolloApiUrl: http://<Internal_Load_Balancer_IP>:4200/graphql

So the

apolloApiUrl

is setting the default for each user of the UI but they can override it from the UI home screen

Ok first of all, thanks for all these helpful directions. I guess let me make note of all this information and digest it. Networking is not my specialty. So I'll need to do some research for more info on a lot of what you said above. But my preliminary thoughts are that this doesn't seem very seamless if the user has to port forward the internal address.

If the apollo api is externally available but limited to some whitelist of IPs then you could set the default

apolloApiUrl

default and the user wouldn’t have to do anything.

So basically, the answer is create a firewall for the external load balancer of the apollo api and limit it to specific IP addresses?

I think that’d be the easiest solution

Okay I will do that then. Makes a lot of sense. Thanks for the suggestion For the UI, I will create an ingress with basic auth. The UI seems to be independent of the other components so nothing should break if I add that security layer to the UI.