https://prefect.io logo
Join Slack
Powered by
Folks, Good evening/afternoon/morning! :slightly_s...
# ask-community
a
Folks, Good evening/afternoon/morning! 🙂 . Does anyone has OpenShift Yaml templates to set up PREFECT ? Due to privacy reason we cant use Prefect Cloud. So need to set up everything on our private OpenShift.. Anyone has Openshift Build , Deploy yaml templates
k
That we do not. We just have the helm chart. Can you use helm with your openshift?
j
OpenShift should support Helm fine (new versions of 4.x actually have a built in graphical wizard too, which is really nice). One caveat is that a lot of our images currently run as root, so you need to make sure that you’re using the anyuid security context constraint for your service account. We haven’t tested it, but I used to be an engineer at Red Hat working on OpenShift and have some experience with it, so I’m happy to help you get things working :)
🙌 1
k
Here is the helm chart
đź‘Ť 1
a
Thanks @Kevin Kho , @jawnsy
d
All the perfect image are rootless now and run on openshift. We are running this setup on our side
đź‘Ť 2
a
I am trying to install Prefect on our private Openshift and getting an error like this. What could be the cause? Could you please help with your thoughts?
j
I guess the error message is saying that you’re not allowed to create a loadbalancer or nodeport service, my guess is that you need to use a ClusterIP and then create a Route or Ingress
a
@jawnsy - you mean to say I have less permission on OCP Project which I trying to install..
j
you probably need to change this to 
type: ClusterIP
and then enable the ingress (or manually create an ingress). Recent versions of OpenShift 4 should understand Ingress and translate it into a Route, but you might need some annotations to control the behavior annotations in the OpenShift doc - https://docs.openshift.com/container-platform/4.9/networking/routes/route-configuration.html#nw-ingress-creating-a-rout[…]ngress_route-configuration https://github.com/PrefectHQ/server/blob/0b64bfd8e6dfc53cddc057575322b3520f1884ea/helm/prefect-server/values.yaml#L219-L223
đź‘Ť 1
a
I changed 2 instances of type:LoadBalancer on values.yaml to type:ClusterIP and deployment went fine - Thanks for that @jawnsy. But seems like not all PODs are up. Are they any minimum resource requirement for Prefect Services - I can see my memory is 100%
@jawnsy: We have an increased resource on our OCP now. But after helm install , seems like postgres POD is not up and I am seeing an error like this. "`create Pod prefect-server-1656368390-postgresql-0 in StatefulSet prefect-server-1656368390-postgresql failed error: pods "prefect-server-1656368390-postgresql-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "pipelines-scc": Forbidden: not usable by user or serviceaccount, provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1001}: 1001 is not an allowed group, spec.containers[0].securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1011190000, 1011199999], provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "rsync-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "aqua-scc": Forbidden: not usable by user or serviceaccount, provider "elasticsearch-scc": Forbidden: not usable by user or serviceaccount, provider "log-collector-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "trident": Forbidden: not usable by user or serviceaccount]"`
j
Did you add your service account to the anyuid security context constraint? If not or if you can’t do that, you will need to modify the chart, and we can’t really help too much with that. It looks like the error message is because OpenShift is not permitting our deployments to run as user 1001 as our Helm charts require. If you want to run under the restricted scc, then you might need to build your own images to ensure that files are readable by the restricted SCC user (e.g. group readable)
There’s some advice here under “Support arbitrary uids” https://docs.openshift.com/container-platform/4.10/openshift_images/create-images.html
a
@jawnsy: Sorry, I havent added any SA - Could you please point me to list of SAs required for me to set up PREFECT. I can give a try!
j
The Helm chart should create service accounts but the service accounts will only be allowed to use the restricted scc by default, and you’d need to grant them anyuid or nonroot. I think the service account is defined here: https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/serviceaccount.yaml You can try running 
oc get sa
in your project/namespace and see what comes up. The docs about SCCs are here: https://docs.openshift.com/container-platform/4.10/authentication/managing-security-context-constraints.html There’s a command you have to run to add the nonroot scc to the service account
đź‘Ť 1
it’s an oc adm command somewhere
a
@jawnsy: I checked with our OCP administrator and they mentioned , "Only the restricted SCC is allowed" - anyuid or noonroot is NOT allowed. So , if I understand your previous statements correctly, then only wayforward is to create our own docker images - Right?
Because using PREFECT helm chart its trying to add 1001 user --
j
Currently, yes. Either build your own images, or you might be able to do it by customizing the Helm installer only, I’m not sure. We also have a professional services team that can make sorts of changes.
đź‘Ť 1
a
@jawnsy: What all services need to have this elevated permissions - 1001 user? Like Hasura, Apollo, Towel , UI etc
Because I can see this requirement only for Hasura
j
I’m not sure if it’s a requirement per se, the default behavior on non-OpenShift installs of Kubernetes is to use the USER from the image (the user of the last layer) when the deployment manifest does not define one. It may work properly with a different user, but we have not tested. Typical things that cause these containers to fail on OpenShift are file permissions (as described in one of the articles I linked earlier in the OpenShift docs), because if we build the image as USER 1001 and mark the file as 700, then we expect it to be readable by the running image, since in most k8s distros, it will run as user 1001. But in OpenShift, we need group permissions or else the UID that the restricted SCC assigns will not be able to read the files, since the container is running as the restricted SCC UID instead of the image layer UID
You can try to see if you can reproduce the problems by running with podman or docker and explicitly setting a different uid, e.g. 
docker run --rm -it -u 123456789:0
(OpenShift runs using the root group and a randomized per-namespace UID)
a
@jawnsy :Well I am trying to get restrictions lifted on our OC project by our admin team, so that it will be a smooth sail, without any major customization on PREFECT services. Curios to know @davzucky - How was your set up on OCP ? Does your platform allow anyuid, non root users ?
@jawnsy I made progress and set up Prefect on OCP and all pods are up and running. Now, when I am trying add new project it's not working or adding projects to the system. What could you be the cause??
j
It might be a database connectivity problem? Hard to say without logs
a
Ok, This is what I am seeing, at the browser console logs. Nothing at POD level.
This is the app : https://prefectuiroute-e11dbf-dev.apps.klab.devops.gov.bc.ca/ . Could you please let me know which log you were mentioning?
j
In the OpenShift console, the logs for the various deployments
a
No errors at POD/container level!!!! . I exposed Openshift routes only for UI and graphql (APOLLO) services - is there any other service that is getting consumed from UI which i need to create public routes
a
sorry Abin, we've shared everything we know about the problem and it's hard to troubleshoot such infrastructure issues via Slack -- if you need more guidance here, I recommend you reach out to a paid support cs@prefect.io
176 Views
Open in Slack
PreviousNext
Powered by