https://prefect.io logo
v

Vincent

02/16/2021, 2:36 PM
Hi. I was wondering if anyone has noticed that some task-runs get resubmitted when running with prefect-cloud/dask. For example, in the screenshot below. I have a task that completed within 51 minutes. However, the reported completion time shows 9 hours. It seems that sometime during the flow, the task run was resubmitted and hit the cached state, which updated the logs. Does anyone have any ideas why the task resubmitted?
j

Jenny

02/16/2021, 3:05 PM
Hi @Vincent - thanks for sharing. I don't have an immediate answer for you but am asking the wider team to see if they have any ideas.
Hi @Vincent - it's likely that Dask resubmitted the work for you - possibly because a worker died. The cached state is probably from Prefect persisting the result of the first run for you.
v

Vincent

02/16/2021, 4:31 PM
That makes a lot of sense. As a feature request, it would great if the cards could capture these retries and report the real runtime of the task. Thanks for the diagnostic!
j

Jenny

02/16/2021, 4:36 PM
@Marvin open "Capture Dask retries and report real runtime of retried tasks"
j

Jenny

02/16/2021, 4:36 PM
@Vincent - Please add any more details of your request in the ticket so we can best track it!
👍 1