Hey! I've a little problem with long running parent flows: There is one parent flow and two sub-flows. The subflows both need between 2 and 30 hours (yes, yes, I know...). Now, quite often, the parent flow enters state "crashed" with State message "Flow run infrastructure exited with non-zero status code -1.". Interestingly, it always happens around runtime of 4 hours. Fortunately, the subflows still continue to run and they also finish. Therefore, it does not really impact the pipeline, because the pipeline simply succeeds - but it's not really nice, that parent flows. Any ideas what could cause the parent to fail? In the logs of the infrastructure (see thread), everything seems fine. (One thing I have to admit: The flows are generating approx. 3 lines of log per second. Is this the problem?). (Second thing to admit: There are a looot of tasks. About 2.000 tasks). prefect 2.0 cloud, v2.7.0 agents. Infrastructure is a kubernetes job. EDIT: I changed to logs and tasks a little bit to have way less logs and tasks. In total there are less than 100 tasks and about 10 logs per minute. But still the same. The parent flows crashed after 4 hours - and the subflows happily continued.

Hi, I m having a smilar issue where I run ML training for a few hours, and deploy it afterwards as subflow. See here : https://prefect-community.slack.com/archives/CL09KU1K7/p1670248970739099

Yup it looks like you both found the right issue to comment on.