Thread
#prefect-community
    Abin Antony

    Abin Antony

    3 months ago
    Folks, Good evening/afternoon/morning! 🙂 . Does anyone has OpenShift Yaml templates to set up PREFECT ? Due to privacy reason we cant use Prefect Cloud. So need to set up everything on our private OpenShift.. Anyone has Openshift Build , Deploy yaml templates
    Kevin Kho

    Kevin Kho

    3 months ago
    That we do not. We just have the helm chart. Can you use helm with your openshift?
    j

    jawnsy

    3 months ago
    OpenShift should support Helm fine (new versions of 4.x actually have a built in graphical wizard too, which is really nice). One caveat is that a lot of our images currently run as root, so you need to make sure that you’re using the anyuid security context constraint for your service account. We haven’t tested it, but I used to be an engineer at Red Hat working on OpenShift and have some experience with it, so I’m happy to help you get things working 😃
    Kevin Kho

    Kevin Kho

    3 months ago
    Here is the helm chart
    Abin Antony

    Abin Antony

    3 months ago
    Thanks @Kevin Kho , @jawnsy
    davzucky

    davzucky

    3 months ago
    All the perfect image are rootless now and run on openshift. We are running this setup on our side
    Abin Antony

    Abin Antony

    3 months ago
    I am trying to install Prefect on our private Openshift and getting an error like this. What could be the cause? Could you please help with your thoughts?
    j

    jawnsy

    3 months ago
    I guess the error message is saying that you’re not allowed to create a loadbalancer or nodeport service, my guess is that you need to use a ClusterIP and then create a Route or Ingress
    Abin Antony

    Abin Antony

    3 months ago
    @jawnsy - you mean to say I have less permission on OCP Project which I trying to install..
    j

    jawnsy

    3 months ago
    you probably need to change this to
    type: ClusterIP
    and then enable the ingress (or manually create an ingress). Recent versions of OpenShift 4 should understand Ingress and translate it into a Route, but you might need some annotations to control the behavior annotations in the OpenShift doc - https://docs.openshift.com/container-platform/4.9/networking/routes/route-configuration.html#nw-ingress-creating-a-rout[…]ngress_route-configuration https://github.com/PrefectHQ/server/blob/0b64bfd8e6dfc53cddc057575322b3520f1884ea/helm/prefect-server/values.yaml#L219-L223
    Abin Antony

    Abin Antony

    3 months ago
    I changed 2 instances of type:LoadBalancer on values.yaml to type:ClusterIP and deployment went fine - Thanks for that @jawnsy. But seems like not all PODs are up. Are they any minimum resource requirement for Prefect Services - I can see my memory is 100%
    @jawnsy: We have an increased resource on our OCP now. But after helm install , seems like postgres POD is not up and I am seeing an error like this. "create Pod prefect-server-1656368390-postgresql-0 in StatefulSet prefect-server-1656368390-postgresql failed error: pods "prefect-server-1656368390-postgresql-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "pipelines-scc": Forbidden: not usable by user or serviceaccount, provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1001}: 1001 is not an allowed group, spec.containers[0].securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1011190000, 1011199999], provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "rsync-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "aqua-scc": Forbidden: not usable by user or serviceaccount, provider "elasticsearch-scc": Forbidden: not usable by user or serviceaccount, provider "log-collector-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "trident": Forbidden: not usable by user or serviceaccount]"
    j

    jawnsy

    2 months ago
    Did you add your service account to the anyuid security context constraint? If not or if you can’t do that, you will need to modify the chart, and we can’t really help too much with that. It looks like the error message is because OpenShift is not permitting our deployments to run as user 1001 as our Helm charts require. If you want to run under the restricted scc, then you might need to build your own images to ensure that files are readable by the restricted SCC user (e.g. group readable)
    There’s some advice here under “Support arbitrary uids” https://docs.openshift.com/container-platform/4.10/openshift_images/create-images.html
    Abin Antony

    Abin Antony

    2 months ago
    @jawnsy: Sorry, I havent added any SA - Could you please point me to list of SAs required for me to set up PREFECT. I can give a try!
    j

    jawnsy

    2 months ago
    The Helm chart should create service accounts but the service accounts will only be allowed to use the restricted scc by default, and you’d need to grant them anyuid or nonroot. I think the service account is defined here: https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/serviceaccount.yaml You can try running
    oc get sa
    in your project/namespace and see what comes up. The docs about SCCs are here: https://docs.openshift.com/container-platform/4.10/authentication/managing-security-context-constraints.html There’s a command you have to run to add the nonroot scc to the service account
    it’s an oc adm command somewhere
    Abin Antony

    Abin Antony

    2 months ago
    @jawnsy: I checked with our OCP administrator and they mentioned , "Only the restricted SCC is allowed" - anyuid or noonroot is NOT allowed. So , if I understand your previous statements correctly, then only wayforward is to create our own docker images - Right?
    Because using PREFECT helm chart its trying to add 1001 user --
    j

    jawnsy

    2 months ago
    Currently, yes. Either build your own images, or you might be able to do it by customizing the Helm installer only, I’m not sure. We also have a professional services team that can make sorts of changes.
    Abin Antony

    Abin Antony

    2 months ago
    @jawnsy: What all services need to have this elevated permissions - 1001 user? Like Hasura, Apollo, Towel , UI etc
    Because I can see this requirement only for Hasura
    j

    jawnsy

    2 months ago
    I’m not sure if it’s a requirement per se, the default behavior on non-OpenShift installs of Kubernetes is to use the USER from the image (the user of the last layer) when the deployment manifest does not define one. It may work properly with a different user, but we have not tested. Typical things that cause these containers to fail on OpenShift are file permissions (as described in one of the articles I linked earlier in the OpenShift docs), because if we build the image as USER 1001 and mark the file as 700, then we expect it to be readable by the running image, since in most k8s distros, it will run as user 1001. But in OpenShift, we need group permissions or else the UID that the restricted SCC assigns will not be able to read the files, since the container is running as the restricted SCC UID instead of the image layer UID
    You can try to see if you can reproduce the problems by running with podman or docker and explicitly setting a different uid, e.g.
    docker run --rm -it -u 123456789:0
    (OpenShift runs using the root group and a randomized per-namespace UID)
    Abin Antony

    Abin Antony

    2 months ago
    @jawnsy :Well I am trying to get restrictions lifted on our OC project by our admin team, so that it will be a smooth sail, without any major customization on PREFECT services. Curios to know @davzucky - How was your set up on OCP ? Does your platform allow anyuid, non root users ?
    @jawnsy I made progress and set up Prefect on OCP and all pods are up and running. Now, when I am trying add new project it's not working or adding projects to the system. What could you be the cause??
    j

    jawnsy

    2 months ago
    It might be a database connectivity problem? Hard to say without logs
    Abin Antony

    Abin Antony

    2 months ago
    Ok, This is what I am seeing, at the browser console logs. Nothing at POD level.
    This is the app : https://prefectuiroute-e11dbf-dev.apps.klab.devops.gov.bc.ca/ . Could you please let me know which log you were mentioning?
    j

    jawnsy

    2 months ago
    In the OpenShift console, the logs for the various deployments
    Abin Antony

    Abin Antony

    2 months ago
    No errors at POD/container level!!!! . I exposed Openshift routes only for UI and graphql (APOLLO) services - is there any other service that is getting consumed from UI which i need to create public routes
    Anna Geller

    Anna Geller

    2 months ago
    sorry Abin, we've shared everything we know about the problem and it's hard to troubleshoot such infrastructure issues via Slack -- if you need more guidance here, I recommend you reach out to a paid support cs@prefect.io