Hey everyone! I am new to prefect and I am looking to implement a POC that mimics an implementation we have going in Airflow. All of our DAGs in Airflow use KubernetesPodOperators + a few hooks for each individual task. What's the best way to emulate this in Prefect? I have a self-hosted Prefect instance going on top of Kubernetes, and I have a Kubernetes work pool going. From what I have been reading/gathering, we can only deploy flows as kubernetes pods, and each individual task gets executed within that pod. Is it possible to run individual tasks as k8s pods? I think trying to emulate this workflow in Prefect is a little weird since we would be spinning up a pod to run a flow that also spins up other pods... all via the work pool pod. I am wondering if anyone has had a similar experience?

At the moment it isn't possible to run individual tasks as pods. The way to imitate the model you're describing with Prefect is to have a parent deployment that runs other deployments via

run_deployment

, which you can use in either a blocking fashion (wait for this deployment to finish, then call it again with different params to run the next deployment) or to run many child deployments in parallel, or a mix of both. Sometimes having an entire separate pod for each unit of work can feel like overkill, so my only additional recommendation is to verify there's a good justification for doing so, like dependency conflicts that necessitate different images, or deadline constraints on compute-heavy workloads that necessitate parallel runs.