ECS Workpool worker continues to crash half way th...
# ask-communityd
datamongus
07/15/2024, 6:11 PMECS Workpool worker continues to crash half way through pipeline. I'm only running 1 flow with a max worker of 4 and 1 subflow with a max worker of 2. Has anyone else experienced poor memory management such as this ?
m
Matthew Bell
07/15/2024, 8:09 PMI recently opened a ticket surrounding memory consumption here, which Prefect staff has looked into extensively. Might be the same issue
d
datamongus
07/15/2024, 8:11 PMThank you, have you found any viable work arounds ?
m
Matthew Bell
07/15/2024, 8:13 PMThere are a few workarounds listed in the thread, but Prefect 3 reduces memory consumption substantially, so I'm just waiting for its release to avoid modifying my code.
A couple easy fixes if you haven't done them though are setting
cache_result_in_memory=Falsepersist_result=Falsed
datamongus
07/15/2024, 8:16 PMAwesome thank you. I'm using Prefect 3 RC now
datamongus
07/16/2024, 6:15 AMJust wanted to say thank you for the advice on setting the two flags above. Either one or both of those flags has made a tremendous improvement to my pipeline. Not only have I not experienced the crashing I was experiencing before, but it has actually increased performance significantly in nearly half the amount of time it was previously. I hope the memory issues are resolved in the RC phase of 3.0, i'll keep digging to see what else I can contribute as well.
m
Matthew Bell
07/16/2024, 2:43 PMHappy to help 🍻
9 Views