Hi everyone My name is Jose and I was learning how to deploy Prefect Community #prefect-kubernetes

Hi everyone!, My name is Jose and I was learning h...

Jose Ignacio Gascon Conde

07/06/2023, 10:13 AM

Hi everyone!, My name is Jose and I was learning how to deploy in prefect. Could anybody give me some tutorial for deploy load works in EKS/Kubernetes?, I am working with prefect-server, prefect-agent in the cluster, but the flows is running inside the prefect-agent and I guess is better that this jobs will be run as Kubernetes Jobs, Sorry for ask, but I dont found documentation for prefect 2. Thx in advance!

Christopher Boyd

07/06/2023, 7:13 PM

Hi Jose, there is quite a bit of documentation for running EKS / Kubernetes - you can check out these resources - https://discourse.prefect.io/t/how-to-create-a-kubernetes-cluster-on-aws-eks/1041 https://github.com/PrefectHQ/prefect-helm If you already have an EKS cluster, you just need to deploy the worker chart to get a worker running in the cluster

Christopher Boyd

07/06/2023, 7:15 PM

From there you can move a local flow to the worker by simply updating the work-pool you are choosing to deploy it to : https://docs.prefect.io/2.10.18/tutorial/deployments/ The single caveat is how your code will get into the container. You can either bake it into the container when you build the image, in which case you don’t need anything for a pull step, and can just point to the local flow on the container filesystem. Alternatively, your flow exists remote (Github, S3, Azure File Storage etc,) in which case your pull step will define how to pull it in

Christopher Boyd

07/06/2023, 7:15 PM

@Marvin - how can I deploy the prefect-helm worker chart

🤯 1

Marvin

07/06/2023, 7:17 PM

Ah, the treasure map has been found. Here's how you can deploy the Prefect Helm worker chart: 1. Prerequisites: Ensure you have Helm and Kubernetes CLI (kubectl) installed and configured on your machine. You also need access to a Kubernetes cluster. Helm Install, Kubectl Install 2. Add Prefect Helm Repository: Add the Prefect Helm repository to your Helm client with the following commands:

Copy code

helm repo add prefect <https://prefecthq.github.io/prefect-helm>
   helm repo update

3. Create a Namespace for Prefect Worker: Create a new namespace in your Kubernetes cluster to deploy the Prefect worker:

Copy code

kubectl create namespace prefect

4. Create a Kubernetes Secret for the API Key: Create a file named

api-key.yaml

with the following contents:

Copy code

apiVersion: v1
   kind: Secret
   metadata:
     name: prefect-api-key
     namespace: prefect
   type: Opaque
   data:
     key: <base64-encoded-api-key>

Replace

<base64-encoded-api-key>

with your Prefect Cloud API key encoded in base64. Apply the

api-key.yaml

file to create the Kubernetes secret:

Copy code

kubectl apply -f api-key.yaml

5. Configure Prefect Worker Values: Create a

values.yaml

file to customize the Prefect worker configuration. Add the following contents to the file:

Copy code

worker:
     cloudApiConfig:
       accountId: <target account ID>
       workspaceId: <target workspace ID>
       config:
         workPool: <target work pool name>

6. Install Prefect Worker Using Helm: Now you can install the Prefect worker using the Helm chart with your custom

values.yaml

file:

Copy code

helm install prefect-worker prefect/prefect-worker --namespace=prefect -f values.yaml

7. Verify Deployment: Check the status of your Prefect worker deployment:

Copy code

kubectl get pods -n prefect

You should see the Prefect worker pod running. You can find more detailed information in the Prefect documentation. And remember, if you're having trouble finding your Prefect Cloud Account ID and Workspace ID, there's a pull request that aims to make this process easier. Ah, the joys of human error. It's always the little things, isn't it?

🙌 1

Open in Slack

Previous Next