Thread
#prefect-server
    r

    Roey Brecher

    1 year ago
    apollo:core-0.14.16 docker memory error: we are occasionally seeing failures with the docker container, the container will sometimes restart itself successfully and sometimes it won’t. the hosting machine have enough memory. Is there any setting we can change in terms of how much memory this container is allocated? I couldn’t find it in the server docker-compose file. the error message:
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Roey Brecher, Did you try increasing the memory like this ? Could you move the stacktrace into the thread to keep the channel compact?
    Also, just making sure you know that Prefect Cloud gives you 10000 free task runs per month, which is more than enough to get started with
    r

    Roey Brecher

    1 year ago
    <--- Last few GCs --->
    [23:0x54a57c0]    80569 ms: Scavenge (reduce) 2043.9 (2050.3) -> 2043.2 (2051.3) MB, 2.0 / 0.0 ms  (average mu = 0.138, current mu = 0.003) allocation failure 
    [23:0x54a57c0]    80574 ms: Scavenge (reduce) 2044.1 (2050.3) -> 2043.4 (2051.3) MB, 2.0 / 0.0 ms  (average mu = 0.138, current mu = 0.003) allocation failure 
    [23:0x54a57c0]    80578 ms: Scavenge (reduce) 2044.3 (2050.3) -> 2043.6 (2051.6) MB, 2.0 / 0.0 ms  (average mu = 0.138, current mu = 0.003) allocation failure 
    <--- JS stacktrace --->
    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
     1: 0xa03530 node::Abort() [node]
     2: 0x94e471 node::FatalError(char const*, char const*) [node]
     3: 0xb7773e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
     4: 0xb77ab7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
     5: 0xd32345  [node]
     6: 0xd32ecf  [node]
     7: 0xd40f5b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
     8: 0xd44b1c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
     9: 0xd131fb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
    10: 0x105919f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
    11: 0x13ff179  [node]
    Aborted (core dumped)
    npm ERR! code ELIFECYCLE
    npm ERR! errno 134
    npm ERR! @ serve: `node dist/index.js`
    npm ERR! Exit status 134
    npm ERR! 
    npm ERR! Failed at the @ serve script.
    npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
    npm ERR! A complete log of this run can be found in:
    npm ERR!     /root/.npm/_logs/2021-06-22T15_25_42_163Z-debug.log
    Kevin Kho

    Kevin Kho

    1 year ago
    Thank you!
    r

    Roey Brecher

    1 year ago
    thanks for the link, we don’t use mac or windows for prefect server, but i’ll look it up. I was wondering if there is a setting in prefect to change this, so we won’t need to do it everytime we restart the server.
    Kevin Kho

    Kevin Kho

    1 year ago
    I think you want to change the maximum docker allows, which is 2GB by default so that you don’t have to worry about setting it
    r

    Roey Brecher

    1 year ago
    there isn’t any such limit on Linux apparently . this means there is a memory leak in the docker container which can’t be fixed with increasing the memory limits. are you aware of that?
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I see. Let me get someone who knows more than me.
    Just to clarify, does this happen when you start up the container or when you’re running a flow?
    r

    Roey Brecher

    1 year ago
    this happens when we run many flows concurrently