Hi everyone. Does anyone have any methods that work for monitoring how much memory is used by flow runs? We use Kubernetes (OpenShift) to run our flows, and what I want is some tool or method that will let me answer questions like, How much memory (RAM) does my flow use on an average run? What's its peak usage? What's the highest peak its hit within the past year? Etc. Curious to hear any method you all use to monitor memory usage of individual flows over time.
✅ 1
z
Zanie
06/12/2023, 7:51 PM
Fwiw we do annotate Kubernetes infrastructure for flow runs with flow / deployment metadata which may be helpful for this use-case.
r
redsquare
06/12/2023, 7:56 PM
Datadog does this for us - at a cost however
j
James Ashby
06/12/2023, 8:01 PM
Ah, thanks to both of these answers. That's helpful.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.