https://prefect.io logo
j

Johan Lundin

04/29/2021, 9:30 AM
Hey Team, I have a Kubernetes config question: I setup a k8s prefect agent running in a few clusters. It is pretty straight forward and the prefect commandline tool will spit out the yaml to kubectl to create a k8s Deployment running the agent. The agent then polls the cloud instance for new jobs and spins up new k8s Jobs/Pods that are running the tasks. Given I have a number of different AWS VPCs which all have AWS VPC Endpoints pointing to the same services but with different IPs mapped to them I would like to assign different HostAlias mappings so that same name in multiple clusters can be assigned to different IP addresses. I know it is possible since I have a cluster configured in this way but the guy setting it up has since left. Example output from kubectl in attached file with ******* marking the place of interest and descriptions. Any pointers are of great interest!!!
k

Kevin Kho

04/29/2021, 2:57 PM
Hey @Johan Lundin! Iā€™m gonna grab someone on the team with more k8s experience
j

Johan Lundin

04/30/2021, 12:58 AM
@Kevin Kho many thanks!!
k

Kevin Kho

04/30/2021, 1:08 AM
Hey @Johan Lundin! Unfortunately, this is quite too specific of an issue (and more k8s than Prefect) for us to help with.
But we encourage from people in the community with experience to chime in
upvote 2
j

Johan Lundin

04/30/2021, 8:35 AM
So answering my own question: The way to do it is to use a KubernetesJobEnvironment and specify a job_spec_file as described at https://docs.prefect.io/orchestration/execution/k8s_job_environment.html#process i.e.
f.environment = KubernetesJobEnvironment(job_spec_file="job_spec.yaml")
šŸ™Œ 2
k

Kevin Kho

04/30/2021, 1:28 PM
Glad you figured it out and thanks for looping back!
šŸ™Œ 1
3 Views