Chu Lục Ninh
04/22/2022, 8:40 AMkubernetes.RunNamespacedJob
, when run a long-running job (in my case half of the day). I got huge memory leak. My suspect is this job use ThreadPoolExecutor
to poll job log every 5s using ReadNamespacedPodLogs
, since ReadNamespacedPodLogs executed in other thread context, its refCount didn't reduce to 0 to trigger GC when we assign new value to read_pod_logs
, and that leads to orphan object and leak the memory
https://github.com/PrefectHQ/prefect/blob/14644fa5fe129b7fa8385d58df82e6f27332b04a/src/prefect/tasks/kubernetes/job.py#L773Anna Geller
04/22/2022, 1:04 PMChu Lục Ninh
04/22/2022, 1:11 PMAnna Geller
04/22/2022, 1:28 PMGlad that the community fixed it in v1.1.0does it mean that upgrading your Server and flows will fix the issue for you?
it is the service we want to run everyday in some time range.when you submit your trades, do you think it must be a long-running service? I could easily imagine having it implemented as parametrized flow and to coordinate multiple trades, you could build it as a flow with multiple tasks or even a flow of flows, each flow for a specific market/sector/trade type submission. This way, you could actually have way more visibility than running it as a long-running
KubernetesPodJob
. Not something you could change immediately, but worth thinking aboutChu Lục Ninh
04/22/2022, 2:12 PMdoes it mean that upgrading your Server and flows will fix the issue for you?
v1.1.0 only fixed losing thread context. The memory leak problem still persist in master branch.
I could easily imagine having it implemented as parametrized flow and to coordinate multiple trades
This is not our purpose. We run the service that receives market feed from the exchange from 9AM to 3PM everyday. After that range of time, the exchange close the connection so we have to gracefully shutdown our service and open it back tomorrow.
Anna Geller
04/22/2022, 3:43 PMChu Lục Ninh
04/23/2022, 2:37 AMAnna Geller
04/23/2022, 1:35 PMChu Lục Ninh
04/23/2022, 2:26 PMAnna Geller
04/23/2022, 2:34 PM