Chu Lục Ninh04/22/2022, 8:40 AM
, when run a long-running job (in my case half of the day). I got huge memory leak. My suspect is this job use
to poll job log every 5s using
, since ReadNamespacedPodLogs executed in other thread context, its refCount didn't reduce to 0 to trigger GC when we assign new value to
, and that leads to orphan object and leak the memory https://github.com/PrefectHQ/prefect/blob/14644fa5fe129b7fa8385d58df82e6f27332b04a/src/prefect/tasks/kubernetes/job.py#L773
Anna Geller04/22/2022, 1:04 PM
Chu Lục Ninh04/22/2022, 1:11 PM
Anna Geller04/22/2022, 1:28 PM
Glad that the community fixed it in v1.1.0does it mean that upgrading your Server and flows will fix the issue for you?
it is the service we want to run everyday in some time range.when you submit your trades, do you think it must be a long-running service? I could easily imagine having it implemented as parametrized flow and to coordinate multiple trades, you could build it as a flow with multiple tasks or even a flow of flows, each flow for a specific market/sector/trade type submission. This way, you could actually have way more visibility than running it as a long-running
. Not something you could change immediately, but worth thinking about
Chu Lục Ninh04/22/2022, 2:12 PM
does it mean that upgrading your Server and flows will fix the issue for you?
v1.1.0 only fixed losing thread context. The memory leak problem still persist in master branch.
I could easily imagine having it implemented as parametrized flow and to coordinate multiple trades
This is not our purpose. We run the service that receives market feed from the exchange from 9AM to 3PM everyday. After that range of time, the exchange close the connection so we have to gracefully shutdown our service and open it back tomorrow.
Anna Geller04/22/2022, 3:43 PM
Chu Lục Ninh04/23/2022, 2:37 AM
Anna Geller04/23/2022, 1:35 PM
Chu Lục Ninh04/23/2022, 2:26 PM
Anna Geller04/23/2022, 2:34 PM