Anders Segerberg

05/02/2023, 5:02 PM
Are there any known issues in prefect 1.0 of failed tasks, when retrying, causing memory leaks (LocalDask executor)?
We've run into recurring issues where, when mapping over a large number of tasks (which otherwise should fit well within memory), if some of them start to fail, and then retry, there will be cascading failures which eventually trigger the gQL server going down, as well as the flow -- and then ultimately the VM we're running on, which runs out of swap space and has other extremely-high memory and CPU usage issues. I recognize this is a rather broad question, but, generally speaking, I'm wondering if there was at any point, known issues in this area, related to resource usage and contention