https://prefect.io logo
Channels
Powered by
j

Jose Ignacio Gascon Conde

07/06/2023, 10:13 AM
Hi everyone!, My name is Jose and I was learning how to deploy in prefect. Could anybody give me some tutorial for deploy load works in EKS/Kubernetes?, I am working with prefect-server, prefect-agent in the cluster, but the flows is running inside the prefect-agent and I guess is better that this jobs will be run as Kubernetes Jobs, Sorry for ask, but I dont found documentation for prefect 2. Thx in advance!
c

Christopher Boyd

07/06/2023, 7:13 PM
Hi Jose, there is quite a bit of documentation for running EKS / Kubernetes - you can check out these resources - https://discourse.prefect.io/t/how-to-create-a-kubernetes-cluster-on-aws-eks/1041 https://github.com/PrefectHQ/prefect-helm If you already have an EKS cluster, you just need to deploy the worker chart to get a worker running in the cluster
From there you can move a local flow to the worker by simply updating the work-pool you are choosing to deploy it to : https://docs.prefect.io/2.10.18/tutorial/deployments/ The single caveat is how your code will get into the container. You can either bake it into the container when you build the image, in which case you don’t need anything for a pull step, and can just point to the local flow on the container filesystem. Alternatively, your flow exists remote (Github, S3, Azure File Storage etc,) in which case your pull step will define how to pull it in
@Marvin - how can I deploy the prefect-helm worker chart
🤯 1
m

Marvin

07/06/2023, 7:17 PM
Ah, the treasure map has been found. Here's how you can deploy the Prefect Helm worker chart: 1. Prerequisites: Ensure you have Helm and Kubernetes CLI (kubectl) installed and configured on your machine. You also need access to a Kubernetes cluster. Helm Install, Kubectl Install 2. Add Prefect Helm Repository: Add the Prefect Helm repository to your Helm client with the following commands:
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
   helm repo update
3. Create a Namespace for Prefect Worker: Create a new namespace in your Kubernetes cluster to deploy the Prefect worker:
Copy code
kubectl create namespace prefect
4. Create a Kubernetes Secret for the API Key: Create a file named 
api-key.yaml
with the following contents:
Copy code
apiVersion: v1
   kind: Secret
   metadata:
     name: prefect-api-key
     namespace: prefect
   type: Opaque
   data:
     key: <base64-encoded-api-key>
Replace 
<base64-encoded-api-key>
with your Prefect Cloud API key encoded in base64. Apply the 
api-key.yaml
file to create the Kubernetes secret:
Copy code
kubectl apply -f api-key.yaml
5. Configure Prefect Worker Values: Create a 
values.yaml
file to customize the Prefect worker configuration. Add the following contents to the file:
Copy code
worker:
     cloudApiConfig:
       accountId: <target account ID>
       workspaceId: <target workspace ID>
       config:
         workPool: <target work pool name>
6. Install Prefect Worker Using Helm: Now you can install the Prefect worker using the Helm chart with your custom 
values.yaml
file:
Copy code
helm install prefect-worker prefect/prefect-worker --namespace=prefect -f values.yaml
7. Verify Deployment: Check the status of your Prefect worker deployment:
Copy code
kubectl get pods -n prefect
You should see the Prefect worker pod running. You can find more detailed information in the Prefect documentation. And remember, if you're having trouble finding your Prefect Cloud Account ID and Workspace ID, there's a pull request that aims to make this process easier. Ah, the joys of human error. It's always the little things, isn't it?
🙌 1
Powered by