Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Andrew Pruchinski
07/25/2022, 3:34 PMHey all. Our flow isn't able to spin up more than 180 jobs. We can't determine the root cause exactly but at a certain point there's a connection aborted and prefect starts to remove jobs and then marks the entire flow as failed. We are using a daskexecutor on the cloud. Looking into the dask repos, we aren't able to set some debug level logging as an attempt to retrieve some info on what’s really causing the
Connection aborted✅ 1
b
Bianca Hoch
07/25/2022, 10:40 PMHello Andrew, would you mind sharing the flow id or an ID of an example flow run that was affected by this?
a
Andrew Pruchinski
07/26/2022, 1:28 PMThank you @Bianca Hoch for the response and sorry for the delay
i believe this is an example of one: https://cloud.prefect.io/flow-run/2bac3da9-a43e-4d4b-80df-3739b0440a88
b
Bianca Hoch
07/26/2022, 1:49 PMNo worries, thanks Andrew!
Hm. That's odd. I can't seem to find this flow run. Would you mind sending over another? Or perhaps the flow ID?
Also, to clarify, could you describe the 180 jobs you are trying to run? (are they tasks, a series of subflows containing different tasks, etc?)
a
Andrew Pruchinski
07/26/2022, 1:58 PMHopefully this will work!
Yes so these are all mapped tasks for a machine learning model. We are experimenting over a bunch of different combinations of hyperparameters and models to determine the best one. The flow should spin up around 2800 tasks so we tried bumping up our max jobs to 1000. The resources are built to scale up as needed using dask and Karpenter
👍 1
b
Bianca Hoch
07/26/2022, 2:22 PMAlright, it looks like we have a few errors of interest in the logs:
{"id":"1b8fe323-632b-4f49-966b-a2a82ba1de3a","name":"k8s-infra","timestamp":"2022-07-21T20:43:39.284953+00:00","message":"Pod prefect-job-a70dec73-mrhrw failed.\n\tContainer 'flow' state: terminated\n\t\tExit Code:: 137\n\t\tReason: OOMKilled"}{"id":"c5d8b50b-5eb6-486c-9266-07e5302a60ad","name":"prefect.CloudFlowRunner","timestamp":"2022-07-21T20:43:56.292591+00:00","message":"Flow run is no longer in a running state; the current state is: <Failed: \"Kubernetes Error: pods ['prefect-job-a70dec73-mrhrw'] failed for this job\">"}It looks like an OOM error due to lack of k8s resources
The final Failed state of the flow was accompanied by the following message:
"Kubernetes Error: pods ['prefect-job-a70dec73-mrhrw'] failed for this job*"*Here's some documentation with a few suggestions on how to handle OOM errors:
https://discourse.prefect.io/t/how-to-allocate-more-memory-or-more-worker-nodes-on-a-per-flow-run-basis/130
https://discourse.prefect.io/t/how-to-ensure-that-my-flow-run-kubernetes-job-has-enough-memory/320
a
Andrew Pruchinski
07/26/2022, 5:03 PMPerfect! Thank you for these resources. Ill be looking into them. Thank you for your help @Bianca Hoch
🎉 1
Do you have any documentation or tips and tricks to handling OOM errors when using KubeCluster? This uses Coiled. Thank you @Bianca Hoch!
b
Bianca Hoch
07/28/2022, 1:38 PMHello Andrew, sorry for the late response. Currently we are a bit light on documentation in regards to handling OOM errors when using KubeCluster. Expect more to come on that going in to the future!
a
Andrew Pruchinski
07/28/2022, 6:45 PMThank you!