Oh dear, it appears you're having memory issues with Prefect. It's a common problem, really. The universe is expanding, galaxies are drifting apart, and yet we can't seem to find enough memory to run our programs. It's all very depressing, but let's see what we can do about it.
It seems that your issue might be related to how Prefect handles memory allocation for running flows. There are a few GitHub issues that have been reported which might be related to your problem:
1.
Enhancement: specify default results storage block: This pull request discusses adding a setting that would allow results to be saved to a specified block by default, unless a different storage path or block is explicitly defined in the flow code.
2.
Kubernetes agent log (perhaps fail?) when jobs are rescheduled: This issue discusses how Kubernetes jobs are automatically rescheduled, which can result in an attempted re-execution of the flow run in another pod. The proposed solution is to add an annotation indicating the Prefect job is not safe to evict.
3.
Subprocess Killed Task is Not Marked as Failed: This issue discusses a scenario where Prefect's subprocess is killed (due to exceeding container limits), but Prefect does not properly mark the task as failed.
4.
Kubernetes flow runs remain running on when k8s pods restart: This issue discusses a scenario where Kubernetes pod restarts/evictions result in flow runs remaining in a Running State despite there not being any jobs running on the Kubernetes cluster.
5.
Task Runs Concurrency slots not released when flow runs in Kubernetes are cancelled: This issue discusses a scenario where if the flow run is cancelled while tasks are in a running state, the concurrency slots used by the tasks are not released.
I'd recommend looking into these issues and their related discussions. They might provide some insight into your problem and potential solutions. If none of these seem to apply, you might want to consider opening a new issue on the Prefect GitHub page.
Remember, in the grand scheme of the universe, this is a small problem. Not that it makes it any less frustrating, of course.