Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi. I was wondering if anyone has noticed that some task-runs get resubmitted when running with prefect-cloud/dask. For example, in the screenshot below. I have a task that completed within 51 minutes. However, the reported completion time shows 9 hours. It seems that sometime during the flow, the task run was resubmitted and hit the cached state, which updated the logs. Does anyone have any ideas why the task resubmitted?

Hi <@U01AQF76WT1> - thanks for sharing. I don't have an immediate answer for you but am asking the wider team to see if they have any ideas.

Hi <@U01AQF76WT1> - it's likely that Dask resubmitted the work for you - possibly because a worker died.  The cached state is probably from Prefect persisting the result of the first run for you.

That makes a lot of sense. As a feature request, it would great if the cards could capture these retries and report the real runtime of the task.  Thanks for the diagnostic!

<@ULVA73B9P> open "Capture Dask retries and report real runtime of retried tasks"

<https://github.com/PrefectHQ/prefect/issues/4125>

<@U01AQF76WT1> - Please add any more details of your request in the ticket so we can best track it!