https://prefect.io logo
Join Slack
Powered by
<@U07J4HZ1PS7> is your worker running as that serv...
# prefect-kubernetes
g
@Fabio Machado is your worker running as that service account?
f
Hey, @Gregory Hunt. Thanks for replying me. Service Account: prefect-service-account And i've started the worker with this command:
Copy code
│     Command:                                                                                                                                                                                │
│       prefect                                                                                                                                                                               │
│       worker                                                                                                                                                                                │
│       start                                                                                                                                                                                 │
│       --pool                                                                                                                                                                                │
│       winkernetes                                                                                                                                                                           │
│     State:          Running
g
Did you give that service account the correct permission in the cluster to create jobs
f
my concern is about the name of the name of the account. Please look:
Copy code
eksctl create iamidentitymapping --cluster XXXXXXX --arn arn:aws:iam::11111111111111:user/prefect-service-account --group manager --username system:serviceaccount:big-data:prefect-service-account
i think i'm did this corectly, but still cant creating things
here my prefect.yaml where i call the SA
Copy code
version: '2.0'
name: fabio-deploy-kube28

description: Deployment do meu fluxo utilizando Kubernetes

work_pool:
  name: winkernetes

infrastructure:
  type: kubernetes-job
  blockName: winker-netes    # Substitua pelo nome do seu bloco Kubernetes
  working_directory: /code
  image: XXXXXXX/dwh:flow
  serviceAccountName: prefect-service-account
  tolerations:
    - key: "name"
      operator: "Equal"
      value: "prefect"
      effect: "NoSchedule"
      tolerationSeconds: 3600  
  resources:
    limits:
      cpu: "1"
      memory: "512Mi"
    requests:
      cpu: "0.5"
      memory: "256Mi"


parameters: {}
  # nome: fabio
  # sobrenome: machado

tags:
- winker

schedule: {}
  # cron: '*/1 * * * *'

# storage:
  # type: remote-file-system
  # blockName: "meu-storage-block"  # Substitua pelo nome do bloco de armazenamento remoto, se necessário

deployments:
- name: test-28
  entrypoint: 
  version: '1.0'
  tags:
  - kubernetes
  - meu-fluxo
  parameters: {}
    # nome: fabio
    # sobrenome: machado
  description:
  work_pool:
    name: winkernetes
    work_queue_name: default
    job_variables:
        env:
            PREFECT_API_URL: <http://prefect-server.big-data.svc.cluster.local:4200/api>
        image: '{{ build-image.image }}'
  schedules: {}
  # - cron: '*/1 * * * *'
    # timezone:
    # day_or: true
    # active: true
  build:
  - prefect_docker.deployments.steps.build_docker_image:
      requires: prefect-docker>=0.3.1
      id: build-image
      dockerfile: ./Dockerfile
      image_name: XXXXX/dwh
      tag: flow
      dockerfile: ./Dockerfile      
  push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.3.1
      image_name: '{{ build-image.image_name }}'
      tag: '{{ build-image.tag }}'
      credentials: '{{ prefect_docker.docker-registry-credentials.docker_registry_creds_name
        }}'
  pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /code
please let me know if im did something wrong (i'm sure about this)
g
How are you running your k8s worker
f
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-work-pool
  namespace: big-data
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-work-pool
  template:
    metadata:
      labels:
        app: prefect-work-pool
    spec:
      serviceAccount: prefect-service-account
      containers:
      - name: prefect-work-pool
        image: XXXXXX/dwh:flow
        env:
        - name: PREFECT_API_URL
          value: <http://prefect-server.big-data.svc.cluster.local:4200/api>
      tolerations:
      - key: "name"
        operator: "Equal"
        value: "prefect"
        effect: "NoSchedule"
this way
im using the community edition, and i have migrated to v3
@Gregory Hunt did you see anything wrong?
is mandatory using a cloud api account to running worker properly?
i want to run as a hosted service, without cloud api key. Because this i've used this deployment above
g
that looks fine, you just need to make sure that 
prefect-service-account
has the correct permissions in you cluster, what are the permission errors you get?
f
Hey @Gregory Hunt. Sorry for my late. I had some issues here. Below the error when i run the deploy:
Copy code
big-data prefect-pool-59d95b5689-dd4nx prefect-work-pool prefect.exceptions.InfrastructureError: Unable to create Kubernetes job: Forbidden: jobs.batch is forbidden: User "system:serviceaccount:big-data:prefect-service-account" cannot create resource "jobs" in API group "batch" in the namespace "default"
But, i've grant permissions on all namespaces, including pods and job creations.
And even i've deleted the iammapping to this account 
system:serviceaccount:big-data:prefect-service-account
, the deployments still referencing to it. I'm so concerned about this.
one update for here. I've deleted the deployment of worker and created a new serviceaccount. Now, i can deploy jobs/pods. But, i got a new error related to this 👇
Copy code
|     result = func(*args)
    |              ^^^^^^^^^^^
    |   File "/usr/local/lib/python3.12/ssl.py", line 917, in do_handshake
    |     self._sslobj.do_handshake()
    | ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1000)
thanks for help until now @Gregory Hunt. I'll back if i need anything more.
g
Hmm I haven’t seen that one
f
@Gregory Hunt morning, dude. I've made a new deployment to test the credentials in case of crash and turn back to get the same error messages. Please, let me know if what i did are correct, please: • I had a rbac with a lot of permissions, including creating jobs/pods. • I created one IAM account to do the iam mapping with eksctl • I created one block with cluster config ◦ I created a kubernetes credential based on cluster config • I put the code on flows to load all this configs, but i still receiving errors.
Copy code
from prefect_aws import AwsCredentials
aws_credentials_block = AwsCredentials.load("dwh-aws")
from prefect.blocks.kubernetes import KubernetesClusterConfig
cluster_config_block = KubernetesClusterConfig.load("winkernetes-prefect")
from prefect_kubernetes.credentials import KubernetesCredentials
kubernetes_credentials = KubernetesCredentials.load("kube-dwh-cred")
I need to inject all of them into the worker pod? Or i understand all concepts in a wrong way?
g
Ok, so how I do it is i leave that blank and I give K8s ServiceAccount that my worker is using the permissions to create pods/jobs in the namespace i use, which is prefect
You also need to give that service account list/get permissions for pods
My worker runs inside my cluster and just leverages internal cluster flows
f
just like me
but i just dont understand why the permissions just applied to the pod
you are using eks, too?
g
I don't use any credential blcks
GCP GKE
but should be pretty similiar
f
i'm totally sure about that
@Gregory Hunt, my friend. Thanks for all help. I found the issue about RBAC and now my jobs are working. I need to make some refactorings, but im not stucked anymore.
g
Woot woot
7 Views
Open in Slack
PreviousNext
Powered by