ECS Workpool worker continues to crash half way through pipeline. I'm only running 1 flow with a max...

datamongus

07/15/2024, 6:11 PM

ECS Workpool worker continues to crash half way through pipeline. I'm only running 1 flow with a max worker of 4 and 1 subflow with a max worker of 2. Has anyone else experienced poor memory management such as this ?

Matthew Bell

07/15/2024, 8:09 PM

I recently opened a ticket surrounding memory consumption here, which Prefect staff has looked into extensively. Might be the same issue

datamongus

07/15/2024, 8:11 PM

Thank you, have you found any viable work arounds ?

Matthew Bell

07/15/2024, 8:13 PM

There are a few workarounds listed in the thread, but Prefect 3 reduces memory consumption substantially, so I'm just waiting for its release to avoid modifying my code. A couple easy fixes if you haven't done them though are setting

cache_result_in_memory=False

and

persist_result=False

on your task / flow

datamongus

07/15/2024, 8:16 PM

Awesome thank you. I'm using Prefect 3 RC now

datamongus

07/16/2024, 6:15 AM

Just wanted to say thank you for the advice on setting the two flags above. Either one or both of those flags has made a tremendous improvement to my pipeline. Not only have I not experienced the crashing I was experiencing before, but it has actually increased performance significantly in nearly half the amount of time it was previously. I hope the memory issues are resolved in the RC phase of 3.0, i'll keep digging to see what else I can contribute as well.

Matthew Bell

07/16/2024, 2:43 PM

Happy to help 🍻

10 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.