Hi everyone, i'm trying to build a pipeline which have the following components:
• Schedule Coordinator: Receive event through a queue and parse the event for scheduling job. We are using github storage to create the flow (to avoid import many dependencies of different workflow into scheduler).
• A workpool in K8s
• Worker running flow in K8s also
Everything is working but we have a problem with the Scheduler Coordinator caused OOM error on K8s even we have tried to increase the memory for the pod, the problem is happened when the schedule coordinator trying to pull github (our github repo size is only 5 MB and we already increase the memory of the pod to 1Gb). Is anyone have this problem before? And beside using github storage to create a flow, is there any different way to create flow without needing to import dependencies of these flows?
Thanks!