e

    Eamon Keane

    2 years ago
    I'm still a little unclear on the kubernetes functionality in prefect (after reading the docs). Can I launch a
    CreateNamespacedPod
    task and will the prefect agent monitor the launched pod's exit status (via kubernetes api) as an indicator of task success/failure?
    Dylan

    Dylan

    2 years ago
    Hi @Eamon Keane! What’s your overall goal? There are many ways to skin the kubernetes cat with Prefect. If I have a better idea of what you’re trying to achieve, I might be able to give better advice. To your specific question, the
    CreateNamespacedPod
    task doesn’t monitor the laurched pod’s exit status. If you want to launch a job as part of a Prefect flow that’s monitored & sends state updates, you can use a Kubernetes agent and something like the
    RemoteEnvironment
    or
    DaskKubernetesEnvironment
    .
    We have an issue open for the specific use case you’ve described: https://github.com/PrefectHQ/prefect/issues/2281
    e

    Eamon Keane

    2 years ago
    thanks Dylan yes that might work. Although that is for job (a kubernetes construct which I think has its own retry logic) as opposed to a bare pod. Basically I have an airflow dag of tasks each of which runs in a kubernetes pod (the pods that are launched have no idea of airflow and would also have no knowledge of prefect).
    Dylan

    Dylan

    2 years ago
    So, if I’m reading this right, you’re trying to convert an airflow dag that needs to run in a specific environment, in this case a kubernetes pod, and Prefect won’t be installed in the launched pod
    Is that the gist?
    Or I should ask, are all of the tasks running in the same pod or different pods?
    e

    Eamon Keane

    2 years ago
    it's a dag that consists exclusively of KubernetesPodOperators and is a common airflow use case. All in different pods. https://airflow.apache.org/docs/stable/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html
    Dylan

    Dylan

    2 years ago
    Sure, but what are you trying to achieve with the Kubernetes pods? Is the fact that the workflow is in kubernetes central to what you’re trying to achieve (i.e. I’m running things inside a specific cluster for security reasons and will be modifying that cluster) or is that because you’re running your workflows on Kubernetes as execution infrastructure (i.e. my flow is trying to achieve something not specifically related to Kubernetes, it’s just running in a K8s cluster)
    e

    Eamon Keane

    2 years ago
    K8s as execution infra. it's fivetran/stitch like elt software that I want so give certain cpu and memory resources to (each task would e.g. sync a full db from mysql to snowflake). I need to loop over a certain number of data sources and launch one pod per data source with a templated entrypoint script. Launching pods and controlling the retry logic in prefect makes more sense than launching jobs imo, but there would be use cases for both. https://github.com/transferwise/pipelinewise
    Dylan

    Dylan

    2 years ago
    Ah interesting