Oliver Götze

    Oliver Götze

    1 year ago
    Hi all, I am encountering a PayLoadTooLargeError from Apollo when logging a large amount of messages in a short period of time:
    apollo_1    | PayloadTooLargeError: request entity too large
    (full traceback in the thread) When this error is thrown, no further log entries will be written for that particular flow run. Logging continues however for other flow runs. I've been able to mitigate this problem by decreasing the number of log messages, i.e. merging similar log messages into a single entry before sending them to logging. Nevertheless I was wondering if there's a way to make the logging more resilient? Maybe Prefect Cloud performs better in such a scenario?
    I'm using Prefect Server 0.15.0 on a VPS with 4 Cores, 8 GB RAM and 128 GB SSD.
    graphql_1   | INFO:     172.22.0.6:50274 - "POST /graphql/ HTTP/1.1" 200 OK
    apollo_1    | PayloadTooLargeError: request entity too large
    apollo_1    |     at readStream (/apollo/node_modules/raw-body/index.js:155:17)
    apollo_1    |     at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
    apollo_1    |     at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
    apollo_1    |     at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
    apollo_1    |     at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
    apollo_1    |     at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
    apollo_1    |     at /apollo/node_modules/express/lib/router/index.js:284:7
    apollo_1    |     at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
    apollo_1    |     at next (/apollo/node_modules/express/lib/router/index.js:275:10)
    apollo_1    |     at cors (/apollo/node_modules/cors/lib/index.js:188:7)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Oliver Götze, we’ll look into this. Does this error only happen in apollo or in the flow run too?
    Oliver Götze

    Oliver Götze

    1 year ago
    Hey @Kevin Kho, thanks! 🙂 The error only happens in Apollo, there is no error in the flow run log. After this error has been thrown by Apollo, logging seems to stop for that flow run, and on GraphQL and the UI you can only get the logs up to that moment. The logs are originating from a task, which is importing another python script via importlib.util. The log messages inside this python script are logged to stdout, and are then picked up by the prefect task because it is configured with the log_stdout=True argument. Simplified example, just in case the log_stdout might be a factor:
    @task(name="handle_transformation_run", log_stdout=True)
    def handle_transformation_run(transformation_job, dpt_instance_path):
    	spec = importlib.util.spec_from_file_location("run_transformation_p1", module_path)
    	transformation_module = importlib.util.module_from_spec(spec)
    	spec.loader.exec_module(transformation_module)
    Kevin Kho

    Kevin Kho

    1 year ago
    Will ask the team about this.
    Oliver Götze

    Oliver Götze

    1 year ago
    That's great, thanks 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    Hi @Kevin Kho, I am getting the same error but I have Apollo deployed using Kubernetes, how can I make this adjustment there?
    Kevin Kho

    Kevin Kho

    1 year ago
    I think you can’t @Fabrice Toussaint because I assume the you used
    helm
    to download the images they are already built there. I think what needs to happen is to
    exec
    into the Apollo pod and edit the source? But that would not help if another pod were created. Oliver here might know more than me. You might need to edit the source and create your own container to pass to helm? It seems very involved to get it to work.
    Oliver Götze

    Oliver Götze

    1 year ago
    Hi @Kevin Kho, thanks so much! 🙂 I'm afraid I won't be of much help when it comes to Kubernetes and helm, as I haven't really used it yet. 😕 but it does sound rather involved