Channels
#prefect-server
Title
r
Roey Brecher
06/22/2021, 3:42 PMapollo:core-0.14.16 docker memory error:
we are occasionally seeing failures with the docker container, the container will sometimes restart itself successfully and sometimes it won’t.
the hosting machine have enough memory.
Is there any setting we can change in terms of how much memory this container is allocated? I couldn’t find it in the server docker-compose file.
the error message:
k
Kevin Kho
06/22/2021, 4:20 PM
Hey @Roey Brecher, Did you try increasing the memory like this ? Could you move the stacktrace into the thread to keep the channel compact?
Also, just making sure you know that Prefect Cloud gives you 10000 free task runs per month, which is more than enough to get started with
r
Roey Brecher
06/22/2021, 4:30 PM Copy code
<--- Last few GCs --->
[23:0x54a57c0] 80569 ms: Scavenge (reduce) 2043.9 (2050.3) -> 2043.2 (2051.3) MB, 2.0 / 0.0 ms (average mu = 0.138, current mu = 0.003) allocation failure
[23:0x54a57c0] 80574 ms: Scavenge (reduce) 2044.1 (2050.3) -> 2043.4 (2051.3) MB, 2.0 / 0.0 ms (average mu = 0.138, current mu = 0.003) allocation failure
[23:0x54a57c0] 80578 ms: Scavenge (reduce) 2044.3 (2050.3) -> 2043.6 (2051.6) MB, 2.0 / 0.0 ms (average mu = 0.138, current mu = 0.003) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa03530 node::Abort() [node]
2: 0x94e471 node::FatalError(char const*, char const*) [node]
3: 0xb7773e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb77ab7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd32345 [node]
6: 0xd32ecf [node]
7: 0xd40f5b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
8: 0xd44b1c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xd131fb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x105919f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x13ff179 [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! @ serve: `node dist/index.js`
npm ERR! Exit status 134
npm ERR!
npm ERR! Failed at the @ serve script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-06-22T15_25_42_163Z-debug.logk
Kevin Kho
06/22/2021, 4:30 PM
Thank you!
r
Roey Brecher
06/22/2021, 4:32 PMthanks for the link, we don’t use mac or windows for prefect server, but i’ll look it up.
I was wondering if there is a setting in prefect to change this, so we won’t need to do it everytime we restart the server.
k
Kevin Kho
06/22/2021, 4:33 PM
I think you want to change the maximum docker allows, which is 2GB by default so that you don’t have to worry about setting it
r
Roey Brecher
06/22/2021, 4:41 PMthere isn’t any such limit on Linux apparently .
this means there is a memory leak in the docker container which can’t be fixed with increasing the memory limits.
are you aware of that?
k
Kevin Kho
06/22/2021, 4:46 PM
Oh I see. Let me get someone who knows more than me.
Just to clarify, does this happen when you start up the container or when you’re running a flow?
r
Roey Brecher
06/23/2021, 9:50 AMthis happens when we run many flows concurrently
2 Views