Hi all, I am looking to use the helm chart to deploy on a AWS EKS cluster. Having little knowledge and spending some time reading through the docs, I was wondering if anyone else has done this and have any notes they can send me so I can attempt to set this up today? Thanks in advance, any advice would be appreciated!

Hi @Si Wong -- I maintain the helm chart, let me know if you have any specific questions. It also may be worth looking at the IAC in https://github.com/PrefectHQ/server/pull/171 -- it's just for Azure/GCP right now but it can give you an idea of what infrastructure needs to be deployed.

Thank you for your response @Zanie, I will take a look and if it's ok with you, can I reach out over slack?

Feel free to reach out in this thread, we prefer that users do not DM us here 🙂

Thanks @Zanie, I've revisited this and managed to get the deployment of Prefect-Server working on an EKS cluster. I also took your advice and deployed a Postgres (RDS) instance and configured a k8s secret to hold the database password. All of that works out the box, thank you! I have ran into another challenge and it's most likely due to my k8s technical debt in some key areas but I have now modified the helm chart by including an ingress template and it looks to work both for the UI and Apollo containers. I changed the service from loadbalancer to cluster IP and using the nginx annotation in place of. I have now enabled the agent and tenant but these containers seem to run in an error loop (see attachment). I've narrowed it down to these Environment variables and not quite sure how to link them locally and would prefer not to hardcode the dns in here. Do you have any ideas how this should be configured?

PREFECT__CLOUD__API: <http://prefect-server-apollo.prefect-harbor:4200/graphql/>

Hmm. So that url is constructed to use the builtin cluster DNS which is

<service-name>-<namespace>

which should be accessible although I'm not sure how that changes once an ingress is added.

Thanks @Zanie I will take a look through this, it's nice to be pointed in the right direction so much appreciation for that. Just a quick question re: the env variable above, does it matter that it's labeled

PREFECT__CLOUD__API

despite using prefect-server?

Yeah

CLOUD

just means "backend api" there 🙂

@Zanie your link helped me understand what needed to be done. I was able to navigate to the

_helpers.tpl

file under the apollo directory for the chart and modify out the port. As I have setup a proxy using nginx. I changed it to the below snippet and it now works as expected with a tenant and agent. Onwards and upwards

{{- define "prefect-server.apollo-api-url" -}}
{{- $host := include "prefect-server.apollo-hostname" . -}}
{{- $port := "4200" -}}
{{ printf "<http://%s>" $host }}
{{- end -}}

Ah that makes sense! If you want to contribute back there is an open issue to add ingress support out of the box :) if you don't have time to finish it perhaps someone else will if you open a PR with your changes

I've never actually opened a PR before 😐

If you want to send me a DM I'd be happy to walk you through it, otherwise nbd 🙂

Hi @Zanie, me again and apologies! So I have an interesting challenge I am trying to solve and looking for advice before I run down a rabbit hole! I have authentication wrapped around the UI but would like to achieve the same setup for the Apollo endpoint. However, if I wrap auth redirect on the Apollo ingress, the UI can no longer connect due to the headers. For some reason I can't reference the Apollo [apolloApiUrl: friendly.dns.com] as a private kubneretes dns variable i.e.

<http://prefect-server-apollo.prefect.svc.cluster.local:80/graphql>

It needs to be a public domain which we can't protect, can you advise how I could potentially link the UI > Apollo endpoint utilising kubernetes backbone network rather than the browser looking for a public facing DNS?

Hey! I don't really have any good advice for you. I'm not an expert on auth patterns in K8s. I presume the private dns variable is not working because the UI running in your user's browser needs to be able to contact the API directly (https://github.com/PrefectHQ/server/issues/212). Generally, Server isn't intended to handle auth and most things you can add will be top-level whereas Cloud has auth all the way down to the database and uses Okta for SSO. Having the UI proxy requests to the API is feasible but would be a significant amount of refactoring in the UI which I don't think our team has the bandwidth for.

Hi @Zanie thank you for all the great information. I work with @Si Wong. You mentioned above that Cloud uses Okta for SSO. We have our own Okta domain. Which makes me think could we configure Prefect UI to point at our Okta domain? So that traffic from the UI to Apollo would have an authorization header attached. I can see the config used for Cloud here that we would need to change? https://github.com/PrefectHQ/ui/blob/master/.env Can you see any unforeseen issues if we attempt this? Thank you again.

Hey @Jonathan Wright -- I don't work on the UI but I'll ping someone on that team.