@Marvin How am I getting the below error when the flow is already started and doesn't create any other subflows...

Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started.
04:53:40 PM
prefect.flow_runs.worker
Job event 'SuccessfulCreate' at 2024-06-03 23:45:20+00:00: Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch
04:53:43 PM
prefect.flow_runs.worker
Pod event 'TriggeredScaleUp' at 2024-06-03 23:49:22+00:00: pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}]
04:53:43 PM
prefect.flow_runs.worker
Pod event 'FailedScheduling' (5 times) at 2024-06-03 23:51:06+00:00: 0/17 nodes are available: 12 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..
04:53:43 PM
prefect.flow_runs.worker
Reported flow run '1977ce58-36f6-4095-8a6c-cd08ffd8e2ab' as crashed: Flow run infrastructure exited with non-zero status code -1.

Your message was too long by 82 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

How am I getting the below error when the flow is already started and doesn't create any other subflows...

```Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started. 045340 PM prefect.flow_runs.worker Job event 'SuccessfulCreate' at 2024-06-03 234520+0000 Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch 045343 PM prefect.flow_runs.worker Pod event 'TriggeredScaleUp' at 2024-06-03 234922+0000 pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}] 045343 PM prefect.flow_runs.worker Pod event 'FailedScheduling' (5 times) at 2024-06-03 235106+0000 0/17 nodes are available: 12 Insufficient nvidia.com/gpu, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes

@Marvin How am I getting the below error when the flow is already started and doesn't create any other subflows...

Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started.
prefect.flow_runs.worker
Job event 'SuccessfulCreate' at 2024-06-03 23:45:20+00:00: Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch
prefect.flow_runs.worker
Pod event 'TriggeredScaleUp' at 2024-06-03 23:49:22+00:00: pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}]
prefect.flow_runs.worker
Pod event 'FailedScheduling' (5 times) at 2024-06-03 23:51:06+00:00: 0/17 nodes are available: 12 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..
prefect.flow_runs.worker
Reported flow run '1977ce58-36f6-4095-8a6c-cd08ffd8e2ab' as crashed: Flow run infrastructure exited with non-zero status code -1.

Your message was too long by 54 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

How am I getting the below error when the flow is already started and doesn't create any other subflows...

```Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started. prefect.flow_runs.worker Job event 'SuccessfulCreate' at 2024-06-03 234520+0000 Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch prefect.flow_runs.worker Pod event 'TriggeredScaleUp' at 2024-06-03 234922+0000 pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}] prefect.flow_runs.worker Pod event 'FailedScheduling' (5 times) at 2024-06-03 235106+0000 0/17 nodes are available: 12 Insufficient nvidia.com/gpu, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful

@Marvin How am I getting the below error when the flow pod is already started and showing progress...

Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started.
prefect.flow_runs.worker
Job event 'SuccessfulCreate' at 2024-06-03 23:45:20+00:00: Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch
prefect.flow_runs.worker
Pod event 'TriggeredScaleUp' at 2024-06-03 23:49:22+00:00: pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}]
prefect.flow_runs.worker
Pod event 'FailedScheduling' (5 times) at 2024-06-03 23:51:06+00:00: 0/17 nodes are available: 12 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

Your message was too long by 1 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

How am I getting the below error when the flow pod is already started and showing progress...

```Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started. prefect.flow_runs.worker Job event 'SuccessfulCreate' at 2024-06-03 234520+0000 Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch prefect.flow_runs.worker Pod event 'TriggeredScaleUp' at 2024-06-03 234922+0000 pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}] prefect.flow_runs.worker Pod event 'FailedScheduling' (5 times) at 2024-06-03 235106+0000 0/17 nodes are available: 12 Insufficient nvidia.com/gpu, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

@Marvin How am I getting the below error when the flow pod is already started and showing progress

Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started.
prefect.flow_runs.worker
Job event 'SuccessfulCreate' at 2024-06-03 23:45:20+00:00: Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch
prefect.flow_runs.worker
Pod event 'TriggeredScaleUp' at 2024-06-03 23:49:22+00:00: pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}]
prefect.flow_runs.worker
Pod event 'FailedScheduling' (5 times) at 2024-06-03 23:51:06+00:00: 0/17 nodes are available: 12 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

Your message was too long by 1 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

How am I getting the below error when the flow pod is already started and showing progress

```Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started. prefect.flow_runs.worker Job event 'SuccessfulCreate' at 2024-06-03 234520+0000 Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch prefect.flow_runs.worker Pod event 'TriggeredScaleUp' at 2024-06-03 234922+0000 pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}] prefect.flow_runs.worker Pod event 'FailedScheduling' (5 times) at 2024-06-03 235106+0000 0/17 nodes are available: 12 Insufficient nvidia.com/gpu, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

@Marvin How am I getting this error when the flow pod is already started and showing progress...

Job 'clip-processing-clip-01-001-0hdiwva2-t65g5': Pod never started.
prefect.flow_runs.worker
Job event 'SuccessfulCreate' at 2024-06-03 23:45:20+00:00: Created pod: clip-processing-clip-01-001-0hdiwva2-t65g5-hgbch
prefect.flow_runs.worker
Pod event 'TriggeredScaleUp' at 2024-06-03 23:49:22+00:00: pod triggered scale-up: [{eks-managed-ondemand-gpu-2x-20240410221932672100000001-9ec7650c-23fb-ae15-a964-710268245289 17->19 (max: 20)}]
prefect.flow_runs.worker
Pod event 'FailedScheduling' (5 times) at 2024-06-03 23:51:06+00:00: 0/17 nodes are available: 12 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/17 nodes are available: 12 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

It seems you're encountering a scheduling issue related to your flow pods in Prefect. The error indicates that the pod failed to start due to insufficient resources and node constraints in your Kubernetes cluster. ### Key Points from the Logs: - Pod never started: This typically means the Kubernetes scheduler couldn't find any suitable nodes to run the pod. - **Insufficient nvidia.com/gpu:** Indicates that none of the available nodes have enough GPU resources. - Node affinity/selector failure: Some nodes didn't match the pod's node affinity or selector criteria. ### Potential Solutions: 1. Check Node Resources: - Ensure you have enough GPU resources and that they are correctly labeled in your Kubernetes cluster. - Verify that the

<http://nvidia.com/gpu|nvidia.com/gpu>

resource is correctly advertised by the nodes.

shell
   kubectl get nodes -o json | jq '.items[].status.allocatable'

2. Review Node Affinity Rules: - Double-check the node affinity rules defined in your pod specification to ensure they're not overly restrictive. Example affinity settings:

yaml
   affinity:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
         - matchExpressions:
           - key: <http://nvidia.com/gpu|nvidia.com/gpu>
             operator: Exists

3. Kubernetes Cluster Scaling: - If using a managed Kubernetes service (like EKS), ensure the cluster can scale to accommodate the demand for GPU nodes. 4. Pod Scheduling Insights: - Investigate pod events and descriptions to get more details on scheduling failures.

shell
   kubectl describe pod <pod-name>
   kubectl get events --sort-by='.metadata.creationTimestamp'

### Relevant Discussions and Issues: - Adding more detail to k8s agent logs when pod never starts: Discusses improving log details for pod startup failures. - Scheduling Failures and Affinity Issues: - Ensure node affinity and tolerations are correctly configured. - Review Kubernetes scheduling documentation for more insights. ### Documentation Links: - Prefect Kubernetes Quickstart - Prefect Deployment Configurations By verifying the above, you should be able to pinpoint and resolve the resource and scheduling constraints preventing your pods from starting.

@Marvin The pod clip-processing-clip-01-001-0hdiwva2-wtnjd was running though, why was another pod trying to spin up mid process

It looks like there are a few potential causes and complications that could explain why a new pod might spin up in the middle of processing. Here’s a step-by-step breakdown of what might be happening: ### Key Findings: 1. Pod Scheduling and Resource Constraints: - The original pod might not have been able to get sufficient resources at some point, causing Kubernetes to try and reschedule or spin up a new one. - The

FailedScheduling

errors indicate that nodes were having trouble meeting the required resources (`nvidia.com/gpu`). 2. Multiple Pods in a Single Flow: - It’s possible that Prefect is trying to auto-scale or create multiple instances due to workload distribution strategies. According to this GitHub issue, autoscaling can sometimes lead to pods being started multiple times, especially in environments like GKE Autopilot. 3. Flow Configuration Insights: - In the Prefect documentation, deployments with work pools and base job templates can lead to multiple pods being created depending on the configuration and tasks at hand. For more details, see the Prefect Deployments documentation. ### Potential Solutions & Further Investigation: 1. Check Configuration and Reconcile Resources: - Ensure that your Kubernetes cluster nodes have sufficient resources and are correctly labeled to meet the demands of your pods. - Re-evaluate node affinity and resource requests in your Kubernetes jobs. 2. Review Flow Setup and Autoscaling: - Review the flow and task configuration to make sure that tasks aren’t unintentionally causing new pods to spin up. - Consider adjusting the autoscaling settings, especially if using GKE Autopilot or similar managed services that handle autoscaling automatically. 3. Pod Event Logs: - Investigate the Kubernetes pod events and logs to get more details on why rescheduling or new pod creation was triggered. Use:

sh
     kubectl get events --sort-by='.metadata.creationTimestamp'
     kubectl describe pod <pod-name>

### Useful Links: - Prefect Deployments Documentation - GitHub Issue on Multiple Pod Spawning - Configuring Node Affinity and Tolerations in Kubernetes If after checking these points the issue persists, more context from Kubernetes logs and Prefect flow definitions might be necessary to troubleshoot further.

@Marvin Do tasks spin up new pods?

In Prefect, tasks themselves do not inherently spin up new pods. Pods are typically tied to the execution of flows, not individual tasks. However, depending on the infrastructure environment and how Prefect is configured to manage workloads, there could be scenarios where it seems like tasks are causing new pods to start. Let’s break down the key points: ### How Prefect Uses Pods: 1. Flow Runs: - A pod is generally created for each flow run when using Kubernetes as the infrastructure. - The pod executes the entire flow, which can consist of multiple tasks. 2. Work Pools and Scaling: - If a flow is configured with a Kubernetes work pool, Prefect might spin up additional pods to manage workload efficiently or due to scaling policies. - For instance, in environments where adaptive scaling is in use (e.g., Dask with Kubernetes), new pods can be created dynamically based on the number of tasks running concurrently and resource demands. 3. Dask Executor: - When using a

DaskExecutor

with Kubernetes, tasks can be distributed across multiple Dask workers, which may run in separate pods. - This ties back to how Dask handles scaling and task distribution rather than Prefect inherently spinning up pods for individual tasks. ### Related Documentation Excerpts: - From Prefect Concepts: Deployments: - Deployments can define how individual flow runs are submitted to infrastructure (source). - From GitHub Issues: - Mapped tasks trigger multiple times on GKE Autopilot: Users have noticed mapped tasks may trigger more instances, especially with auto-scaling infrastructure. ### Task Execution Strategy: - Mapped Tasks: - Running tasks in a mapped manner (

task.map()

) can distribute the workload across multiple workers, which in turn can cause multiple pods to be started if done via Dask or similar executors. - This is particularly prevalent in complex or large flows where

ConcurrentTaskRunner

is used. ### Summary: - Tasks themselves do not spin up new pods directly. - Flows associated with pod management can create new pods, particularly influenced by configuration, scaling policies, and the deployment setup. - Using mappers and distributed execution frameworks like Dask can lead to seeing more pod activity if the executor scales out pods to handle the task workload. For deeper insight into a particular behavior in your setup, ensure to check the deployment and execution configuration, and consider how adaptive scaling may be affecting pod creation.