Please see a brief of the issue below   We have a flow which runs on a daily basis and was facing the following issue on a day-to-day:   The flow runs a task that checks if data is available in some s3 locations - if it is not available it will generate the data and then generate some artifacts (s3 artifacts not prefect artifacts) related to the data.   The task run successfully generates the data for the missing s3 locations but runs out of memory when attempting to generate the artifacts related to the data.   Dask then attempts to restart the worker, which results in restarting the task run. Given the data is now available in the s3 locations, the task run will skip generating artifacts and as such the task and flow run will complete successfully.    The consequence of this is we are missing these artifacts and given this was a silent failure we only found out about this two weeks later when we tried to fetch these artifacts.   My question is:  The flow has version locking enabled. Why did the version locking not throw an error to ensure the task runs only once? Are there any other settings we can use to ensure this is raised to error ? Is the current recommendation to configure dask to avoid restarts ?

(Happy to share links to the flow/flow run or task run via DM if need be)

I’m wondering whether instead of version locking, something like this would be helpful: • before you generate the artifacts, you have a task that checks if those artifacts already exist. If so, you raise ENDRUN or FAILED depending on whether you want to retry that task on failure or not • raising FAILED signal respects retries. But if you don’t want the task to retry, you can use ENDRUN

To your questions though, flow version locking is to stop a completed task from running twice. In the case you are describing, I think the first pass is not finishing due to the OOM. The worker is dying so the final state is not reported to Prefect. This is why version locking doesn’t stop another run from triggering. To prevent the restarts, you need to look at configuring the

nanny

service, but I’m not finding specific docs about that

It is true that perhaps splitting this into separate tasks would help for this specific case but I am trying to better understand how task version locking is working

On the Prefect side, you can maybe set up an Automation that sends a message when the Flow is taking too long?

Maybe you can use a Nanny Plugin? Docs here. And then you can put your message there? I have not dug into these myself.