Been having a bit of trouble with Dask crashing recently, and I came across this old thread from a few months ago. I think my issue is very similar to the one described, where a large number of tasks being submitted at once causes the flow to crash. Has there been any updates on this and/or a simple way to solve this?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
