Thread
#prefect-community
    Prabin Mehta

    Prabin Mehta

    1 year ago
    does anybody have any doc to follow for deploying prefect in kubernetes cluster?
    Marko Herkaliuk

    Marko Herkaliuk

    1 year ago
    Prabin Mehta

    Prabin Mehta

    1 year ago
    @*Marko Herkaliuk* Thanks for this. Can this be connected to prefect cloud?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Prabin Mehta, you're trying to connect prefect server to cloud?
    m

    Mariia Kerimova

    1 year ago
    Hello Prabin! Marko shared a helm chart for deploying Prefect Server, which is different from Prefect Cloud. For deploying your flows in Kubernetes cluster, you can use KubernetesRun. The minimal example of using Kubernetes run config:
    from prefect import task, Flow
    from prefect.run_configs import KubernetesRun
    from prefect.storage import Docker
    
    @task
    def say_hello(log_stdout=True):
        print("Hello, world!")
    
    with Flow("Test") as flow:
        flow.add_task(say_hello())
    
    flow.storage = Docker(registry_url="xxx", image_name="my-kube-testflow")
    flow.run_config = KubernetesRun()
    
    if __name__ == "__main__":
        flow.register(project_name="Training")
    Here you can find information about Kubernetes agent you need to run in order to deploy the flows in your cluster.
    Prabin Mehta

    Prabin Mehta

    1 year ago
    @Mariia Kerimova Thanks. will check that out.
    @Kevin Kho I am able to connect locally. right now trying to connect to the prefect cloud with flows running in Kubernetes cluster. I am using github as storage and trying to use KubernetesRun.
    @Mariia Kerimova I have tried running flows using KubernetesRun. I am using the below deployment file for running the Kubernetes agent.
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: prefect-agent
      name: prefect-agent
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prefect-agent
      template:
        metadata:
          labels:
            app: prefect-agent
        spec:
          containers:
          - args:
            - prefect agent kubernetes start
            command:
            - /bin/bash
            - -c
            env:
            - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
              value: ddd---ddddddddddd
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: NAMESPACE
              value: prefect
            - name: IMAGE_PULL_SECRETS
              value: ''
            - name: PREFECT__CLOUD__AGENT__LABELS
              value: '[]'
            - name: JOB_MEM_REQUEST
              value: ''
            - name: JOB_MEM_LIMIT
              value: ''
            - name: JOB_CPU_REQUEST
              value: ''
            - name: JOB_CPU_LIMIT
              value: ''
            - name: IMAGE_PULL_POLICY
              value: ''
            - name: SERVICE_ACCOUNT_NAME
              value: ''
            - name: PREFECT__BACKEND
              value: cloud
            - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
              value: http://:8080
            - name: EXTRA_PIP_PACKAGES
              value: 'chardet==3.0.2 prefect[snowflake]'
            image: prefecthq/prefect:latest-python3.7
            imagePullPolicy: Always
            livenessProbe:
              failureThreshold: 2
              httpGet:
                path: /api/health
                port: 8080
              initialDelaySeconds: 40
              periodSeconds: 40
            name: agent
    ---
    apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
    kind: Role
    metadata:
      name: prefect-agent-rbac
      namespace: prefect
    rules:
    - apiGroups:
      - batch
      - extensions
      resources:
      - jobs
      verbs:
      - '*'
    - apiGroups:
      - ''
      resources:
      - events
      - pods
      verbs:
      - '*'
    ---
    apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
    kind: RoleBinding
    metadata:
      name: prefect-agent-rbac
      namespace: prefect
    roleRef:
      apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
      kind: Role
      name: prefect-agent-rbac
    subjects:
    - kind: ServiceAccount
      name: default
    But still getting my pods having the msg,
    Event: 'Started' on pod 'prefect-job-0460be1c-6p8v6'
    	Message: Started container flow-container
    and still, the jobs are not executed. Let me know your thoughts. (edited)
    m

    Mariia Kerimova

    1 year ago
    Did you set any labels on your flow? Can you check your agent in Prefect Cloud UI dashboard? Is it healthy? You can paste a redacted flow you're registering, it can help to debug faster.