https://prefect.io logo
Join Slack
Powered by
does anybody have any doc to follow for deploying ...
# ask-community
p
does anybody have any doc to follow for deploying prefect in kubernetes cluster?
m
p
@*Marko Herkaliuk* Thanks for this. Can this be connected to prefect cloud?
k
Hi @Prabin Mehta, you're trying to connect prefect server to cloud?
m
Hello Prabin! Marko shared a helm chart for deploying Prefect Server, which is different from Prefect Cloud. For deploying your flows in Kubernetes cluster, you can use KubernetesRun. The minimal example of using Kubernetes run config:
Copy code
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
from prefect.storage import Docker

@task
def say_hello(log_stdout=True):
    print("Hello, world!")

with Flow("Test") as flow:
    flow.add_task(say_hello())

flow.storage = Docker(registry_url="xxx", image_name="my-kube-testflow")
flow.run_config = KubernetesRun()

if __name__ == "__main__":
    flow.register(project_name="Training")
Here you can find information about Kubernetes agent you need to run in order to deploy the flows in your cluster.
p
@Mariia Kerimova Thanks. will check that out.
@Kevin Kho I am able to connect locally. right now trying to connect to the prefect cloud with flows running in Kubernetes cluster. I am using github as storage and trying to use KubernetesRun.
@Mariia Kerimova I have tried running flows using KubernetesRun. I am using the below deployment file for running the Kubernetes agent.
Copy code
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ddd---ddddddddddd
        - name: PREFECT__CLOUD__API
          value: <https://api.prefect.io>
        - name: NAMESPACE
          value: prefect
        - name: IMAGE_PULL_SECRETS
          value: ''
        - name: PREFECT__CLOUD__AGENT__LABELS
          value: '[]'
        - name: JOB_MEM_REQUEST
          value: ''
        - name: JOB_MEM_LIMIT
          value: ''
        - name: JOB_CPU_REQUEST
          value: ''
        - name: JOB_CPU_LIMIT
          value: ''
        - name: IMAGE_PULL_POLICY
          value: ''
        - name: SERVICE_ACCOUNT_NAME
          value: ''
        - name: PREFECT__BACKEND
          value: cloud
        - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
          value: http://:8080
        - name: EXTRA_PIP_PACKAGES
          value: 'chardet==3.0.2 prefect[snowflake]'
        image: prefecthq/prefect:latest-python3.7
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /api/health
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 40
        name: agent
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-agent-rbac
  namespace: prefect
rules:
- apiGroups:
  - batch
  - extensions
  resources:
  - jobs
  verbs:
  - '*'
- apiGroups:
  - ''
  resources:
  - events
  - pods
  verbs:
  - '*'
---
apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
kind: RoleBinding
metadata:
  name: prefect-agent-rbac
  namespace: prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-agent-rbac
subjects:
- kind: ServiceAccount
  name: default
But still getting my pods having the msg,
Copy code
Event: 'Started' on pod 'prefect-job-0460be1c-6p8v6'
	Message: Started container flow-container
and still, the jobs are not executed. Let me know your thoughts. (edited)
m
Did you set any labels on your flow? Can you check your agent in Prefect Cloud UI dashboard? Is it healthy? You can paste a redacted flow you're registering, it can help to debug faster.
3 Views
Open in Slack
PreviousNext
Powered by