Thread
#prefect-server
    m

    Martin Felder

    11 months ago
    Hi, we recently upgraded to Prefect 0.15.6, and now I'm getting these weird errors
    apollo_1   | PayloadTooLargeError: request entity too large
    apollo_1   |     at readStream (/apollo/node_modules/raw-body/index.js:155:17)
    apollo_1   |     at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
    apollo_1   |     at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
    apollo_1   |     at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
    apollo_1   |     at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
    apollo_1   |     at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
    apollo_1   |     at /apollo/node_modules/express/lib/router/index.js:284:7
    apollo_1   |     at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
    apollo_1   |     at next (/apollo/node_modules/express/lib/router/index.js:275:10)
    apollo_1   |     at cors (/apollo/node_modules/cors/lib/index.js:188:7)
    every 2 seconds or so. The server UI also does not seem to be running. Does anyone have an idea what this means?
    Michael Adkins

    Michael Adkins

    11 months ago
    Hey Martin, we're investigating this internally. I'll try to get back to you soon.
    Zach Angell

    Zach Angell

    11 months ago
    hi @Martin Felder, are you able to tell from the logs what request is causing this error?
    m

    Martin Felder

    11 months ago
    Thank you for having a look at this! Not sure which logs you are talking about? Just the commandline output? I attach that and some other pertinent information.
    Since we had to reinstall our external Postgres server, the database is a clean slate. Same effect happens if I leave out the
    --external-postgres
    option, so that's not it apparently. There are no flows or agents registered yet.
    Zach Angell

    Zach Angell

    11 months ago
    Thanks for sending the logs!
    There are no flows or agents registered yet.
    Hmm okay that's good to know. I'm trying to figure out what's hitting the Prefect Server API with large requests πŸ€” I'll try to reproduce on my end
    A fresh install of
    prefect
    0.15.6 +
    prefect server start
    works as expected for me. Is there something else that could be hitting your server api or any other unique features of your server setup?
    m

    Martin Felder

    11 months ago
    Not that I know of - I think I'll try a fresh install, too. The configuration file looks ok to you for a UI visible from the outside? We are always unsure in which sections we have to specify the IP address.
    Zach Angell

    Zach Angell

    11 months ago
    Hmm just based on the logs it looks like something in the UI configuration is off:
    :space_invader::space_invader::space_invader: UI running at localhost:8080 :space_invader::space_invader::space_invader:
    Maybe the
    PayloadTooLargeError
    is just a red herring
    m

    Martin Felder

    11 months ago
    Ok, you were right - it was a red herring! Apparently there were some LocalAgents running while the DB shut down, and something went awry. After killing the agent and restarting the server, the warnings disappear! Sorry for leading you astray!
    But... the UI is still not visible from the outside 😞 With the
    config.toml
    as stated above and adding the
    --expose
    flag to
    prefect server start
    , which I understand I'm supposed to do at 0.15.5+, I'm still getting
    ui_1       | Missing the PREFECT_SERVER__BASE_URL environment variable.  Using default                                                                                                        
    ui_1       | πŸ‘ΎπŸ‘ΎπŸ‘Ύ UI running at localhost:8080 πŸ‘ΎπŸ‘ΎπŸ‘Ύ
    Setting
    export PREFECT_SERVER__BASE_URL=<http://172.16.18.70>
    changes nothing, neither does adding
    -v master -uv master
    to the server start. Strangely enough, the expose flag does not appear when I run
    perfect server start --help
    , although I seem to be using the correct executable:
    $ conda env list
    # conda environments:
    #
    base                     /opt/miniconda
    ws_prefect            *  /opt/miniconda/envs/ws_prefect
    
    $ which prefect
    /opt/miniconda/envs/ws_prefect/bin/prefect
    
    $ conda list | grep prefect
    # packages in environment at /opt/miniconda/envs/ws_prefect:
    prefect                   0.15.6             pyhd8ed1ab_0    conda-forge
    What am I doing wrong?
    Zach Angell

    Zach Angell

    11 months ago
    Ah looking in the code the
    UI running at localhost:8080
    message is also a red herring, it doesn't log the actual UI url there
    Are you able to see the ui if you run
    prefect server start --expose
    ?
    m

    Martin Felder

    11 months ago
    Ha! Indeed I can! Sorry I haven't tried that in my red herring fixation... Thank you very much, problem solved!
    Zach Angell

    Zach Angell

    11 months ago
    Nice! πŸ₯³