Hey all. Our flow isn't able to spin up more than ...
# prefect-communitya
Andrew Pruchinski
07/25/2022, 3:34 PMHey all. Our flow isn't able to spin up more than 180 jobs. We can't determine the root cause exactly but at a certain point there's a connection aborted and prefect starts to remove jobs and then marks the entire flow as failed. We are using a daskexecutor on the cloud. Looking into the dask repos, we aren't able to set some debug level logging as an attempt to retrieve some info on what’s really causing the
Connection aborted✅ 1
b
Bianca Hoch
07/25/2022, 10:40 PMHello Andrew, would you mind sharing the flow id or an ID of an example flow run that was affected by this?
a
Andrew Pruchinski
07/26/2022, 1:28 PMThank you @Bianca Hoch for the response and sorry for the delay
Andrew Pruchinski
07/26/2022, 1:28 PMi believe this is an example of one: https://cloud.prefect.io/flow-run/2bac3da9-a43e-4d4b-80df-3739b0440a88
b
Bianca Hoch
07/26/2022, 1:49 PMNo worries, thanks Andrew!
Bianca Hoch
07/26/2022, 1:55 PMHm. That's odd. I can't seem to find this flow run. Would you mind sending over another? Or perhaps the flow ID?
Bianca Hoch
07/26/2022, 1:56 PMAlso, to clarify, could you describe the 180 jobs you are trying to run? (are they tasks, a series of subflows containing different tasks, etc?)
a
Andrew Pruchinski
07/26/2022, 1:58 PMAndrew Pruchinski
07/26/2022, 1:58 PMHopefully this will work!
Andrew Pruchinski
07/26/2022, 2:04 PMYes so these are all mapped tasks for a machine learning model. We are experimenting over a bunch of different combinations of hyperparameters and models to determine the best one. The flow should spin up around 2800 tasks so we tried bumping up our max jobs to 1000. The resources are built to scale up as needed using dask and Karpenter
👍 1
b
Bianca Hoch
07/26/2022, 2:22 PMAlright, it looks like we have a few errors of interest in the logs:
Bianca Hoch
07/26/2022, 2:23 PM{"id":"1b8fe323-632b-4f49-966b-a2a82ba1de3a","name":"k8s-infra","timestamp":"2022-07-21T20:43:39.284953+00:00","message":"Pod prefect-job-a70dec73-mrhrw failed.\n\tContainer 'flow' state: terminated\n\t\tExit Code:: 137\n\t\tReason: OOMKilled"}Bianca Hoch
07/26/2022, 2:24 PM{"id":"c5d8b50b-5eb6-486c-9266-07e5302a60ad","name":"prefect.CloudFlowRunner","timestamp":"2022-07-21T20:43:56.292591+00:00","message":"Flow run is no longer in a running state; the current state is: <Failed: \"Kubernetes Error: pods ['prefect-job-a70dec73-mrhrw'] failed for this job\">"}Bianca Hoch
07/26/2022, 2:26 PMIt looks like an OOM error due to lack of k8s resources
Bianca Hoch
07/26/2022, 2:29 PMThe final Failed state of the flow was accompanied by the following message:
"Kubernetes Error: pods ['prefect-job-a70dec73-mrhrw'] failed for this job*"*Bianca Hoch
07/26/2022, 2:33 PMHere's some documentation with a few suggestions on how to handle OOM errors:
https://discourse.prefect.io/t/how-to-allocate-more-memory-or-more-worker-nodes-on-a-per-flow-run-basis/130
https://discourse.prefect.io/t/how-to-ensure-that-my-flow-run-kubernetes-job-has-enough-memory/320
a
Andrew Pruchinski
07/26/2022, 5:03 PMPerfect! Thank you for these resources. Ill be looking into them. Thank you for your help @Bianca Hoch
🎉 1
Andrew Pruchinski
07/27/2022, 3:57 PMDo you have any documentation or tips and tricks to handling OOM errors when using KubeCluster? This uses Coiled. Thank you @Bianca Hoch!
b
Bianca Hoch
07/28/2022, 1:38 PMHello Andrew, sorry for the late response. Currently we are a bit light on documentation in regards to handling OOM errors when using KubeCluster. Expect more to come on that going in to the future!
a
Andrew Pruchinski
07/28/2022, 6:45 PMThank you!
3 Views