Hello :wave: question. I have a mapped task with 8 runs in it. 2/8 tasks are hung (the dask-workers ...

Alexander van Eck

10/14/2021, 2:56 PM

Hello 👋 question. I have a mapped task with 8 runs in it. 2/8 tasks are hung (the dask-workers aren’t responding anymore) but I don’t want to cancel the entire flow and re-run it. What is supposed to happen when I set the task state of these 2/8 to Resume? They are then automatically progressed to Submitted. but never picked up by a worker for execution?

Evan Sutherland

10/14/2021, 2:59 PM

Hey @Alexander van Eck, great question. Let me look into that for you

Zach Angell

10/14/2021, 3:03 PM

Hey @Alexander van Eck a few clarifying questions • Do you want these two tasks to re-run? • Is the flow still running? If so, are any other tasks in the flow still running?

Zach Angell

10/14/2021, 3:05 PM

If the dask workers aren't responding, the 2 task runs will be submitted to the workers but will never be able to execute

upvote 1

Alexander van Eck

10/14/2021, 3:06 PM

I think I’ve just solved it!

Alexander van Eck

10/14/2021, 3:06 PM

I set the two task runs to Resume, then killed those dask-worker processes manually. Dask then spun up two new workers and the tasks started executing. 🙌

🙌 3

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.