I'm still a little unclear on the kubernetes functionality in prefect (after reading the docs). Can I launch a

CreateNamespacedPod

task and will the prefect agent monitor the launched pod's exit status (via kubernetes api) as an indicator of task success/failure?

Hi @Eamon Keane! What’s your overall goal? There are many ways to skin the kubernetes cat with Prefect. If I have a better idea of what you’re trying to achieve, I might be able to give better advice. To your specific question, the

CreateNamespacedPod

task doesn’t monitor the laurched pod’s exit status. If you want to launch a job as part of a Prefect flow that’s monitored & sends state updates, you can use a Kubernetes agent and something like the

RemoteEnvironment

or

DaskKubernetesEnvironment

.

thanks Dylan yes that might work. Although that is for job (a kubernetes construct which I think has its own retry logic) as opposed to a bare pod. Basically I have an airflow dag of tasks each of which runs in a kubernetes pod (the pods that are launched have no idea of airflow and would also have no knowledge of prefect).

So, if I’m reading this right, you’re trying to convert an airflow dag that needs to run in a specific environment, in this case a kubernetes pod, and Prefect won’t be installed in the launched pod

it's a dag that consists exclusively of KubernetesPodOperators and is a common airflow use case. All in different pods. https://airflow.apache.org/docs/stable/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html

Sure, but what are you trying to achieve with the Kubernetes pods? Is the fact that the workflow is in kubernetes central to what you’re trying to achieve (i.e. I’m running things inside a specific cluster for security reasons and will be modifying that cluster) or is that because you’re running your workflows on Kubernetes as execution infrastructure (i.e. my flow is trying to achieve something not specifically related to Kubernetes, it’s just running in a K8s cluster)

K8s as execution infra. it's fivetran/stitch like elt software that I want so give certain cpu and memory resources to (each task would e.g. sync a full db from mysql to snowflake). I need to loop over a certain number of data sources and launch one pod per data source with a templated entrypoint script. Launching pods and controlling the retry logic in prefect makes more sense than launching jobs imo, but there would be use cases for both. https://github.com/transferwise/pipelinewise

Ah interesting