Hi, We have been using prefect 1 (self hosted) in ...
# prefect-getting-started
s
Hi, We have been using prefect 1 (self hosted) in our current product for entire data pipeline for last couple of years. Yep, we are way past the EOL on prefect 1 and we now need to move to prefect 2. The main use case we have for prefect was to launch custom spark jobs on k8s using ShellTask. So on every flow launch the prefect-agent would create a k8s Job based on RunConfig and this k8s Job would then execute the flow spawning ShellTasks. As I understand, In prefect 2 to achieve the similar thing we need to, 1. Create a k8s workpool 2. Create a deployment with the k8s workpool 3. Start the worker 4. The worker would launch the Job which will then execute the flow If my understanding above is correct, I have few questions/clarifications 1. In my understanding, k8s workpool is somewhat like defining a k8s Job template (prefect 1 run_config). Am I correct? 2. I already have the CI workflows to build the docker images and push them to ACR. I don't want them to be part of deployment. https://docs.prefect.io/latest/guides/deployment/kubernetes/#define-a-prefect-deployment are all sections mandatory ? Lastly, is it possible for me to try these on local k8s setup where 1. prefect server is launched locally using
prefect server start
2. docker image is build locally 3. register k8s workpool and provide docker image which needs to be picked from local registry Any samples/blogs if available would be really helpful