Hi we recently upgraded to Prefect 0 15 6 and now I m gettin Prefect Community #prefect-server

Hi, we recently upgraded to Prefect 0.15.6, and no...

Martin Felder

10/13/2021, 4:31 PM

Hi, we recently upgraded to Prefect 0.15.6, and now I'm getting these weird errors

apollo_1   | PayloadTooLargeError: request entity too large

apollo_1   |     at readStream (/apollo/node_modules/raw-body/index.js:155:17)

apollo_1   |     at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)

apollo_1   |     at read (/apollo/node_modules/body-parser/lib/read.js:77:3)

apollo_1   |     at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)

apollo_1   |     at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)

apollo_1   |     at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)

apollo_1   |     at /apollo/node_modules/express/lib/router/index.js:284:7

apollo_1   |     at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)

apollo_1   |     at next (/apollo/node_modules/express/lib/router/index.js:275:10)

apollo_1   |     at cors (/apollo/node_modules/cors/lib/index.js:188:7)

every 2 seconds or so. The server UI also does not seem to be running. Does anyone have an idea what this means?

Zanie

10/13/2021, 5:04 PM

Hey Martin, we're investigating this internally. I'll try to get back to you soon.

Zach Angell

10/13/2021, 6:08 PM

hi @Martin Felder, are you able to tell from the logs what request is causing this error?

Martin Felder

10/14/2021, 8:55 AM

Thank you for having a look at this! Not sure which logs you are talking about? Just the commandline output? I attach that and some other pertinent information.

👍 1

Martin Felder

10/14/2021, 8:59 AM

Since we had to reinstall our external Postgres server, the database is a clean slate. Same effect happens if I leave out the

--external-postgres

option, so that's not it apparently. There are no flows or agents registered yet.

Zach Angell

10/14/2021, 1:20 PM

Thanks for sending the logs!

There are no flows or agents registered yet.

Hmm okay that's good to know. I'm trying to figure out what's hitting the Prefect Server API with large requests 🤔 I'll try to reproduce on my end

Zach Angell

10/14/2021, 1:33 PM

A fresh install of

prefect

0.15.6 +

prefect server start

works as expected for me. Is there something else that could be hitting your server api or any other unique features of your server setup?

Martin Felder

10/18/2021, 6:51 AM

Not that I know of - I think I'll try a fresh install, too. The configuration file looks ok to you for a UI visible from the outside? We are always unsure in which sections we have to specify the IP address.

Zach Angell

10/18/2021, 1:42 PM

Hmm just based on the logs it looks like something in the UI configuration is off:

:space_invader::space_invader::space_invader: UI running at localhost:8080 :space_invader::space_invader::space_invader:

Maybe the

PayloadTooLargeError

is just a red herring

Martin Felder

10/20/2021, 10:04 AM

Ok, you were right - it was a red herring! Apparently there were some LocalAgents running while the DB shut down, and something went awry. After killing the agent and restarting the server, the warnings disappear! Sorry for leading you astray!

Martin Felder

10/20/2021, 10:10 AM

But... the UI is still not visible from the outside 😞 With the

config.toml

as stated above and adding the

--expose

flag to

prefect server start

, which I understand I'm supposed to do at 0.15.5+, I'm still getting

Copy code

ui_1       | Missing the PREFECT_SERVER__BASE_URL environment variable.  Using default                                                                                                        
ui_1       | 👾👾👾 UI running at localhost:8080 👾👾👾

Setting

export PREFECT_SERVER__BASE_URL=<http://172.16.18.70>

changes nothing, neither does adding

-v master -uv master

to the server start. Strangely enough, the expose flag does not appear when I run

perfect server start --help

, although I seem to be using the correct executable:

Copy code

$ conda env list
# conda environments:
#
base                     /opt/miniconda
ws_prefect            *  /opt/miniconda/envs/ws_prefect

$ which prefect
/opt/miniconda/envs/ws_prefect/bin/prefect

$ conda list | grep prefect
# packages in environment at /opt/miniconda/envs/ws_prefect:
prefect                   0.15.6             pyhd8ed1ab_0    conda-forge

What am I doing wrong?

Zach Angell

10/20/2021, 1:28 PM

Ah looking in the code the

UI running at localhost:8080

message is also a red herring, it doesn't log the actual UI url there

Zach Angell

10/20/2021, 1:28 PM

Are you able to see the ui if you run

prefect server start --expose

Martin Felder

10/20/2021, 2:27 PM

Ha! Indeed I can! Sorry I haven't tried that in my red herring fixation... Thank you very much, problem solved!

Zach Angell

10/20/2021, 4:41 PM

Nice! 🥳

2 Views

Open in Slack

Previous Next