apollo core 0 14 16 docker memory error we are occasionally

Question

apollo core 0 14 16 docker memory error we are occasionally seeing failures with the docker container the container will sometimes restart itself successfully and sometimes it won t the hosting machine have enough memory Is there any setting we can change in terms of how much memory this container is allocated I couldn t find it in the server docker compose file the error message

Kevin Kho · Answer

Hey < Roey Brecher> Did you try increasing the memory like Could you move the stacktrace into the thread to keep the channel compact

Kevin Kho · Answer

Also just making sure you know that Prefect Cloud gives you 10000 free task runs per month which is more than enough to get started with

Roey Brecher · Answer

``` lt Last few GCs gt 23 0x54a57c0 80569 ms Scavenge reduce 2043 9 2050 3 gt 2043 2 2051 3 MB 2 0 0 0 ms average mu = 0 138 current mu = 0 003 allocation failure 23 0x54a57c0 80574 ms Scavenge reduce 2044 1 2050 3 gt 2043 4 2051 3 MB 2 0 0 0 ms average mu = 0 138 current mu = 0 003 allocation failure 23 0x54a57c0 80578 ms Scavenge reduce 2044 3 2050 3 gt 2043 6 2051 6 MB 2 0 0 0 ms average mu = 0 138 current mu = 0 003 allocation failure lt JS stacktrace gt FATAL ERROR Ineffective mark compacts near heap limit Allocation failed JavaScript heap out of memory 1 0xa03530 node Abort node 2 0x94e471 node FatalError char const char const node 3 0xb7773e v8 Utils ReportOOMFailure v8 internal Isolate char const bool node 4 0xb77ab7 v8 internal V8 FatalProcessOutOfMemory v8 internal Isolate char const bool node 5 0xd32345 node 6 0xd32ecf node 7 0xd40f5b v8 internal Heap CollectGarbage v8 internal AllocationSpace v8 internal GarbageCollectionReason v8 GCCallbackFlags node 8 0xd44b1c v8 internal Heap AllocateRawWithRetryOrFailSlowPath int v8 internal AllocationType v8 internal AllocationOrigin v8 internal AllocationAlignment node 9 0xd131fb v8 internal Factory NewFillerObject int bool v8 internal AllocationType v8 internal AllocationOrigin node 10 0x105919f v8 internal Runtime AllocateInYoungGeneration int unsigned long v8 internal Isolate node 11 0x13ff179 node Aborted core dumped npm ERR code ELIFECYCLE npm ERR errno 134 npm ERR serve `node dist index js` npm ERR Exit status 134 npm ERR npm ERR Failed at the serve script npm ERR This is probably not a problem with npm There is likely additional logging output above npm ERR A complete log of this run can be found in npm ERR root npm logs 2021 06 22T15 25 42 163Z debug log```

Kevin Kho · Answer

Thank you

Roey Brecher · Answer

thanks for the link we don t use mac or windows for prefect server but i ll look it up I was wondering if there is a setting in prefect to change this so we won t need to do it everytime we restart the server

Kevin Kho · Answer

I think you want to change the maximum docker allows which is 2GB by default so that you don t have to worry about setting it

Roey Brecher · Answer

there isn t any such limit on Linux apparently this means there is a memory leak in the docker container which can t be fixed with increasing the memory limits are you aware of that

Kevin Kho · Answer

Oh I see Let me get someone who knows more than me

Kevin Kho · Answer

Just to clarify does this happen when you start up the container or when you re running a flow

Roey Brecher · Answer

this happens when we run many flows concurrently