https://prefect.io logo
Join Slack
Powered by
Hi all, I am encountering a PayLoadTooLargeError f...
# ask-community
o
Hi all, I am encountering a PayLoadTooLargeError from Apollo when logging a large amount of messages in a short period of time: 
apollo_1    | PayloadTooLargeError: request entity too large
(full traceback in the thread) When this error is thrown, no further log entries will be written for that particular flow run. Logging continues however for other flow runs. I've been able to mitigate this problem by decreasing the number of log messages, i.e. merging similar log messages into a single entry before sending them to logging. Nevertheless I was wondering if there's a way to make the logging more resilient? Maybe Prefect Cloud performs better in such a scenario?
I'm using Prefect Server 0.15.0 on a VPS with 4 Cores, 8 GB RAM and 128 GB SSD.
Copy code
graphql_1   | INFO:     172.22.0.6:50274 - "POST /graphql/ HTTP/1.1" 200 OK
apollo_1    | PayloadTooLargeError: request entity too large
apollo_1    |     at readStream (/apollo/node_modules/raw-body/index.js:155:17)
apollo_1    |     at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
apollo_1    |     at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
apollo_1    |     at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
apollo_1    |     at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
apollo_1    |     at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
apollo_1    |     at /apollo/node_modules/express/lib/router/index.js:284:7
apollo_1    |     at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
apollo_1    |     at next (/apollo/node_modules/express/lib/router/index.js:275:10)
apollo_1    |     at cors (/apollo/node_modules/cors/lib/index.js:188:7)
k
Hey @Oliver Götze, we’ll look into this. Does this error only happen in apollo or in the flow run too?
o
Hey @Kevin Kho, thanks! 🙂 The error only happens in Apollo, there is no error in the flow run log. After this error has been thrown by Apollo, logging seems to stop for that flow run, and on GraphQL and the UI you can only get the logs up to that moment. The logs are originating from a task, which is importing another python script via importlib.util. The log messages inside this python script are logged to stdout, and are then picked up by the prefect task because it is configured with the log_stdout=True argument. Simplified example, just in case the log_stdout might be a factor:
Copy code
@task(name="handle_transformation_run", log_stdout=True)
def handle_transformation_run(transformation_job, dpt_instance_path):
	spec = importlib.util.spec_from_file_location("run_transformation_p1", module_path)
	transformation_module = importlib.util.module_from_spec(spec)
	spec.loader.exec_module(transformation_module)
k
Will ask the team about this.
o
That's great, thanks 🙂
k
👍 1
f
Hi @Kevin Kho, I am getting the same error but I have Apollo deployed using Kubernetes, how can I make this adjustment there?
k
I think you can’t @Fabrice Toussaint because I assume the you used 
helm
to download the images they are already built there. I think what needs to happen is to 
exec
into the Apollo pod and edit the source? But that would not help if another pod were created. Oliver here might know more than me. You might need to edit the source and create your own container to pass to helm? It seems very involved to get it to work.
o
Hi @Kevin Kho, thanks so much! 🙂 I'm afraid I won't be of much help when it comes to Kubernetes and helm, as I haven't really used it yet. 😕 but it does sound rather involved
10 Views
Open in Slack
PreviousNext
Powered by