Thread
#prefect-community
    j

    Johan Lundin

    1 year ago
    Hey Team, I have a Kubernetes config question: I setup a k8s prefect agent running in a few clusters. It is pretty straight forward and the prefect commandline tool will spit out the yaml to kubectl to create a k8s Deployment running the agent. The agent then polls the cloud instance for new jobs and spins up new k8s Jobs/Pods that are running the tasks. Given I have a number of different AWS VPCs which all have AWS VPC Endpoints pointing to the same services but with different IPs mapped to them I would like to assign different HostAlias mappings so that same name in multiple clusters can be assigned to different IP addresses. I know it is possible since I have a cluster configured in this way but the guy setting it up has since left. Example output from kubectl in attached file with ******* marking the place of interest and descriptions. Any pointers are of great interest!!!
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Johan Lundin! Iā€™m gonna grab someone on the team with more k8s experience
    j

    Johan Lundin

    1 year ago
    @Kevin Kho many thanks!!
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Johan Lundin! Unfortunately, this is quite too specific of an issue (and more k8s than Prefect) for us to help with.
    But we encourage from people in the community with experience to chime in
    j

    Johan Lundin

    1 year ago
    So answering my own question: The way to do it is to use a KubernetesJobEnvironment and specify a job_spec_file as described at https://docs.prefect.io/orchestration/execution/k8s_job_environment.html#process i.e.
    f.environment = KubernetesJobEnvironment(job_spec_file="job_spec.yaml")
    Kevin Kho

    Kevin Kho

    1 year ago
    Glad you figured it out and thanks for looping back!