https://prefect.io logo
Title
j

Jax

03/19/2023, 12:09 AM
Hi, i’m new to Prefect and it looks promising. Basically i’m trying to build a standard toolkit for deploying pipelines in production, with Kubernetes. I read the documentation but some points weren’t clear to me. 1. Does Prefect support custom reusable components and a library of it for users to choose from? (E.g. Supports some form of definition and allows checking of interfaces) 2. Does Prefect support building pipelines using UI? Thus far it seems very pythonic. 3. Say i would want to deploy a pipeline with Prefect and using Kubernetes as the infra, can i ask Prefect to do it on a single k8s node? Reason for doing that is to create a low latency pipeline. 4. Does prefect deployment allow HPA?
r

redsquare

03/19/2023, 6:09 AM
1. Have you got an example? 2. Not as such no 3. Sure just use affinity/nodeSelector in your job manifest 4. Prefect is orthogonal to HPA, it creates jobs for the flows to run on a single pod, you could spread the workload out using dask runners or do you mean HPA in terms of the agent?