Hello! I’m running a pretty extensive pipeline on a large EC2 machine (we’re still trying to get setup to port it to prefect cloud), and have some bottleneck tasks in the middle that collect the results of a bunch of previous mapped tasks and do some calculations across the whole dataset. The bottleneck tasks don’t manage to succeed, but they don’t explicitly “fail”, either, earlier tasks just re-run (collecting the checkpointed results) and then the bottleneck task starts again, but again does not finish… I can’t figure out what the problem is, I can run through the task outside of prefect if I load all the data, so I’m not sure if it’s a silent memory issue on the worker running the large task, or some time-out issue with the scheduler, or something else I’m not thinking of. Has anyone else run into similar behavior and have a suggestion for what to do to diagnose and/or fix the problem?

Sounds like a weird scenario. I can’t really tell from the description. Could you mention more about “don’t explicitly fail?“?

That’s what I thought. I’m not sure what else to say, unfortunately, because prefect doesn’t say anything, it just starts rescheduling upstream tasks, which complete with the cached results, and restarts the task that it never gets past.

I think it sounds like Dask workers are dying causing Dask to spin new workers and restart work. And Dask is retrying the work added to Prefect retries?