Hi everyone After installation step by step below the dashbo

Question

Hi everyone After installation step by step below the dashboard shows up but under API Status notes Network error Failed to fetch and asks whether the graphql url field in the config file was set After going through the docs I m not sure under which TOML section this field should be let alone the correct value Furthermore not sure this is really the underlying issue because there are some connection refused errors about Postgres in the server logs upon startup First time user grateful for any leads

Zachary Hughes · Answer

Hi < Burkhard Hoeckendorf> let s start with the Postgres errors you mentioned and work our way up Do you mind sharing the error logs you re seeing there

Burkhard Hoeckendorf · Answer

Sure just 1 sec

Burkhard Hoeckendorf · Answer

gt prefect backend server Backend switched to server gt prefect server start Pulling postgres done Pulling hasura done Pulling graphql done Pulling apollo done Pulling scheduler done Pulling ui done Creating network prefect server with the default driver Creating tmp postgres 1 done Creating tmp hasura 1 done Creating tmp graphql 1 done Creating tmp scheduler 1 done Creating tmp apollo 1 done Creating tmp ui 1 done Attaching to tmp postgres 1 tmp hasura 1 tmp graphql 1 tmp apollo 1 tmp scheduler 1 tmp ui 1 hasura 1 | type startup timestamp 2020 07 21T19 43 20 646+0000 level info detail kind server configuration info live query options batch size 100 refetch delay 1 transaction isolation ISOLATION LEVEL READ COMMITTED plan cache options plan cache size 100 enabled log types http log websocket log startup webhook log server host HostAny enable allowlist false log level info auth hook mode null use prepared statements true unauth role null stringify numeric types false enabled apis metadata graphql config pgdump enable telemetry true enable console true auth hook null jwt secret null cors config allowed origins disabled false ws read cookie null console assets dir null admin secret set false port 3000 hasura 1 | type startup timestamp 2020 07 21T19 43 20 646+0000 level info detail kind postgres connection info retries 1 database url postgres 5432 prefect server postgres 1 | The files belonging to this database system will be owned by user postgres postgres 1 | This user must also own the server process postgres 1 | postgres 1 | The database cluster will be initialized with locale en US utf8 postgres 1 | The default database encoding has accordingly been set to UTF8 postgres 1 | The default text search configuration will be set to english postgres 1 | postgres 1 | Data page checksums are disabled graphql 1 | graphql 1 | Running Alembic migrations apollo 1 | apollo 1 | gt serve apollo apollo 1 | gt node dist index js apollo 1 | graphql 1 | INFO alembic runtime migration Context impl PostgresqlImpl ui 1 | space invader space invader space invader UI running at localhost 8080 space invader space invader space invader postgres 1 | postgres 1 | fixing permissions on existing directory var lib postgresql data ok scheduler 1 | 2020 07 21 19 43 29 448 INFO prefect server Scheduler | Scheduler will start after an initial delay of 295 seconds hasura 1 | internal could not connect to server Connection refused n tIs the server running on host postgres 172 29 0 2 and accepting n tTCP IP connections on port 5432 n path $ error connection error code postgres error graphql 1 | INFO alembic runtime migration Will assume transactional DDL postgres 1 | creating subdirectories ok graphql 1 | INFO alembic runtime migration Running upgrade gt 723da8052cbd create extensions graphql 1 | graphql 1 | Could not upgrade the database graphql 1 | Error psycopg2 errors UniqueViolation duplicate key value violates unique constraint pg extension name index graphql 1 | DETAIL Key extname = pgcrypto already exists graphql 1 | graphql 1 | SQL graphql 1 | CREATE EXTENSION IF NOT EXISTS pgcrypto graphql 1 | CREATE EXTENSION IF NOT EXISTS pg trgm graphql 1 | graphql 1 | Background on this error at postgres 1 | selecting default max connections 100 postgres 1 | selecting default shared buffers 128MB postgres 1 | selecting default timezone Etc UTC postgres 1 | selecting dynamic shared memory implementation posix postgres 1 | creating configuration files ok postgres 1 | running bootstrap script ok postgres 1 | performing post bootstrap initialization ok postgres 1 | syncing data to disk postgres 1 | WARNING enabling trust authentication for local connections postgres 1 | You can change this by editing pg hba conf or using the option A or postgres 1 | auth local and auth host the next time you run initdb postgres 1 | ok postgres 1 | postgres 1 | Success You can now start the database server using postgres 1 | postgres 1 | pg ctl D var lib postgresql data l logfile start postgres 1 | postgres 1 | waiting for server to start 2020 07 21 19 43 20 038 UTC 46 LOG listening on Unix socket var run postgresql s PGSQL 5432 postgres 1 | 2020 07 21 19 43 20 448 UTC 47 LOG database system was shut down at 2020 07 21 19 43 17 UTC postgres 1 | 2020 07 21 19 43 20 627 UTC 46 LOG database system is ready to accept connections postgres 1 | done postgres 1 | server started postgres 1 | CREATE DATABASE postgres 1 | postgres 1 | postgres 1 | usr local bin docker entrypoint sh ignoring docker entrypoint initdb d postgres 1 | postgres 1 | 2020 07 21 19 43 23 726 UTC 46 LOG received fast shutdown request postgres 1 | waiting for server to shut down 2020 07 21 19 43 23 796 UTC 46 LOG aborting any active transactions postgres 1 | 2020 07 21 19 43 23 800 UTC 46 LOG background worker logical replication launcher PID 53 exited with exit code 1 postgres 1 | 2020 07 21 19 43 23 800 UTC 48 LOG shutting down postgres 1 | 2020 07 21 19 43 24 292 UTC 46 LOG database system is shut down postgres 1 | done postgres 1 | server stopped postgres 1 | postgres 1 | PostgreSQL init process complete ready for start up postgres 1 | postgres 1 | 2020 07 21 19 43 24 420 UTC 1 LOG listening on IPv4 address 0 0 0 0 port 5432 postgres 1 | 2020 07 21 19 43 24 420 UTC 1 LOG listening on IPv6 address port 5432 postgres 1 | 2020 07 21 19 43 24 642 UTC 1 LOG listening on Unix socket var run postgresql s PGSQL 5432 postgres 1 | 2020 07 21 19 43 25 113 UTC 64 LOG database system was shut down at 2020 07 21 19 43 24 UTC postgres 1 | 2020 07 21 19 43 25 115 UTC 65 FATAL the database system is starting up postgres 1 | 2020 07 21 19 43 25 172 UTC 1 LOG database system is ready to accept connections postgres 1 | 2020 07 21 19 43 28 479 UTC 73 ERROR duplicate key value violates unique constraint pg extension name index postgres 1 | 2020 07 21 19 43 28 479 UTC 73 DETAIL Key extname = pgcrypto already exists postgres 1 | 2020 07 21 19 43 28 479 UTC 73 STATEMENT postgres 1 | CREATE EXTENSION IF NOT EXISTS pgcrypto postgres 1 | CREATE EXTENSION IF NOT EXISTS pg trgm postgres 1 | apollo 1 | 2020 07 21T19 43 31 141Z Error fetching GraphQL health FetchError request to failed reason connect ECONNREFUSED 172 29 0 3 4201 apollo 1 | 2020 07 21T19 43 31 144Z Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 Could not safely build a schema Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 apollo 1 | 2020 07 21T19 43 31 148Z apollo 1 | Trying again in 3 seconds apollo 1 | apollo 1 | 2020 07 21T19 43 34 158Z Error fetching GraphQL health FetchError request to failed reason connect ECONNREFUSED 172 29 0 3 4201 apollo 1 | 2020 07 21T19 43 34 158Z Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 Could not safely build a schema Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 apollo 1 | 2020 07 21T19 43 34 158Z apollo 1 | Trying again in 3 seconds apollo 1 | apollo 1 | 2020 07 21T19 43 37 165Z Error fetching GraphQL health FetchError request to failed reason connect ECONNREFUSED 172 29 0 3 4201 apollo 1 | 2020 07 21T19 43 37 165Z Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 Could not safely build a schema Error Could not safely build a schema apollo 1 | at safelyBuildSchema apollo dist index js 129 11 apollo 1 | at process tickCallback internal process next tick js 68 7 apollo 1 | 2020 07 21T19 43 37 165Z apollo 1 | Trying again in 3 seconds apollo 1 | apollo 1 | 2020 07 21T19 43 40 188Z Building schema apollo 1 | 2020 07 21T19 43 41 125Z Building schema complete apollo 1 | Server ready at rocket

Burkhard Hoeckendorf · Answer

Let me know if there is a better way than pasting them here

Zachary Hughes · Answer

Do you see this issue deterministically each time you spin up server or is this the first time you re encountering it

Burkhard Hoeckendorf · Answer

Each time

Burkhard Hoeckendorf · Answer

Here is how I installed Here is my setup gt conda create n prefect python gt conda activate prefect gt conda install c conda forge docker compose prefect Docker v19 03 12 docker and containerd are running Here is the conda env Name Version Build Channel libgcc mutex 0 1 main appdirs 1 4 3 py 1 conda forge asn1crypto 1 3 0 py38h32f6830 1 conda forge attrs 19 3 0 py 0 conda forge brotlipy 0 7 0 py38h1e0a361 1000 conda forge ca certificates 2020 6 20 hecda079 0 conda forge cached property 1 5 1 py 0 conda forge certifi 2020 6 20 py38h32f6830 0 conda forge cffi 1 14 0 py38he30daa8 1 chardet 3 0 4 py38h32f6830 1006 conda forge click 7 1 2 pyh9f0ad1d 0 conda forge cloudpickle 1 5 0 py 0 conda forge croniter 0 3 30 py 0 conda forge cryptography 3 0 py38h766eaa4 0 conda forge cytoolz 0 10 1 py38h516909a 0 conda forge dask core 2 21 0 py 0 conda forge distributed 2 21 0 py38h32f6830 0 conda forge distro 1 5 0 pyh9f0ad1d 0 conda forge docker compose 1 26 2 py38h32f6830 0 conda forge docker py 4 2 2 py38h32f6830 0 conda forge docker pycreds 0 4 0 py 0 conda forge dockerpty 0 4 1 py 0 conda forge docopt 0 6 2 py 1 conda forge heapdict 1 0 1 py 0 conda forge idna 2 10 pyh9f0ad1d 0 conda forge importlib metadata 1 7 0 py38h32f6830 0 conda forge importlib metadata 1 7 0 0 conda forge jsonschema 3 2 0 py38h32f6830 1 conda forge ld impl linux 64 2 33 1 h53a641e 7 libedit 3 1 20191231 h14c3975 1 libffi 3 3 he6710b0 2 libgcc ng 9 1 0 hdf63c60 0 libstdcxx ng 9 1 0 hdf63c60 0 marshmallow 3 7 1 py 0 conda forge marshmallow oneofschema 2 0 1 py 0 conda forge msgpack python 1 0 0 py38hbf85e49 1 conda forge mypy extensions 0 4 3 py38h32f6830 1 conda forge ncurses 6 2 he6710b0 1 openssl 1 1 1g h516909a 0 conda forge packaging 20 4 pyh9f0ad1d 0 conda forge pendulum 2 1 1 py38h32f6830 0 conda forge pip 20 1 1 py38 1 prefect 0 12 4 py 0 conda forge psutil 5 7 2 py38h1e0a361 0 conda forge pycparser 2 20 pyh9f0ad1d 2 conda forge pyopenssl 19 1 0 py 1 conda forge pyparsing 2 4 7 pyh9f0ad1d 0 conda forge pyrsistent 0 16 0 py38h1e0a361 0 conda forge pysocks 1 7 1 py38h32f6830 1 conda forge python 3 8 3 hcff3b4d 2 python box 4 2 3 py 0 conda forge python dateutil 2 8 1 py 0 conda forge python dotenv 0 14 0 pyh9f0ad1d 0 conda forge python slugify 4 0 1 pyh9f0ad1d 0 conda forge python abi 3 8 1 cp38 conda forge pytz 2020 1 pyh9f0ad1d 0 conda forge pytzdata 2020 1 pyh9f0ad1d 0 conda forge pyyaml 5 3 1 py38h1e0a361 0 conda forge readline 8 0 h7b6447c 0 requests 2 24 0 pyh9f0ad1d 0 conda forge ruamel yaml 0 16 6 py38h1e0a361 1 conda forge ruamel yaml clib 0 2 0 py38h1e0a361 1 conda forge setuptools 49 2 0 py38 0 six 1 15 0 pyh9f0ad1d 0 conda forge sortedcontainers 2 2 2 pyh9f0ad1d 0 conda forge sqlite 3 32 3 h62c20be 0 tabulate 0 8 7 pyh9f0ad1d 0 conda forge tblib 1 6 0 py 0 conda forge text unidecode 1 3 py 0 conda forge texttable 1 6 2 py 0 conda forge tk 8 6 10 hbc83047 0 toml 0 10 1 pyh9f0ad1d 0 conda forge toolz 0 10 0 py 0 conda forge tornado 6 0 4 py38h1e0a361 1 conda forge unidecode 1 1 1 py 0 conda forge urllib3 1 25 9 py 0 conda forge websocket client 0 57 0 py38h32f6830 1 conda forge wheel 0 34 2 py38 0 xz 5 2 5 h7b6447c 0 yaml 0 2 5 h516909a 0 conda forge zict 2 0 0 py 0 conda forge zipp 3 1 0 py 0 conda forge zlib 1 2 11 h7b6447c 3

Zachary Hughes · Answer

Thanks for sharing Can you confirm that you re able to register a flow I ask because it looks like the Postgres error could be a bit of a red herring and I m trying to determine if this is a Postgres issue or an issue communicating between the UI and the API

Burkhard Hoeckendorf · Answer

I see Give me a minute to try

Burkhard Hoeckendorf · Answer

I can not register a flow Here is the python output of flow register gt Result check OK gt Flow <http localhost 8080 flow 6c37641a e35c 48a6 a594 93596955d0b1> But the flow does not appear on the dashboard and the url doesn t exist on the server

Burkhard Hoeckendorf · Answer

I don t think here are any new lines on the server console

Zachary Hughes · Answer

Solid this is actually super helpful The fact that you got a link back indicates that you were able to persist a flow but that your UI isn t properly configured to point to your API Are you running everything on one machine

Burkhard Hoeckendorf · Answer

Yes form the same Conda env

nicholas · Answer

< Burkhard Hoeckendorf> do you mind taking a screenshot of your API Status tile That ll help understand where it s trying to connect

nicholas · Answer

Perfect thank you Can you visit <http localhost 4200 graphql> in your browser

Burkhard Hoeckendorf · Answer

No

nicholas · Answer

What does visiting that link show Nothing at all or a message about a `GET` request

Burkhard Hoeckendorf · Answer

Nothing at all

nicholas · Answer

Ok that s helpful that means the API isn t where it would normally be Looking back at your stack trace it looks like it s at <http 0 0 0 0 4200 graphql> are you able to visit that link in your browser

Burkhard Hoeckendorf · Answer

Same result I thought 0 0 0 0 is just localhost but externally accessible I m doing everything on the same machine

Burkhard Hoeckendorf · Answer

I also didn t change the default config

nicholas · Answer

They re a little different but it looks like either way the API isn t available at that endpoint either so whatever network Prefect Server is running on isn t available on your local machine

Burkhard Hoeckendorf · Answer

Ok got it Thanks for your help diagnosing this How do we get to the bottom of it

nicholas · Answer

Are you running Prefect Server in an EC2 or some other remote environment

Burkhard Hoeckendorf · Answer

No it is all on my local machine

Burkhard Hoeckendorf · Answer

Ok I just noticed something funny Even after `colda activate prefect` `which python` points to the system python and with this interpreter I can t even `import prefect`

Burkhard Hoeckendorf · Answer

colda = conda

Burkhard Hoeckendorf · Answer

I can successfully execute a script that uses prefect without server UI but when I start python myself I can not import any prefect modules

Burkhard Hoeckendorf · Answer

Not sure how much my local config matters though considering the server is containerized

nicholas · Answer

Hm interesting That shouldn t impact anything in this case since this is a series of networked Docker containers

Burkhard Hoeckendorf · Answer

ok

nicholas · Answer

I m at a little bit of a loss if you re unable to reach the API through your browser by visiting the apollo address localhost 4200 or 0 0 0 0 4200 I m not surprised the UI isn t able to connect cause it s trying that same address

Burkhard Hoeckendorf · Answer

I m not aware of anything that would prevent accessing this address

Burkhard Hoeckendorf · Answer

Is there any way or would it be worth double checking the service is actually running

nicholas · Answer

Well as long as you ve not exited the process you started with `prefect server start` it should be running still

Burkhard Hoeckendorf · Answer

Ok

nicholas · Answer

you can try to restart it by sigterming the process and starting it again

nicholas · Answer

that might give you some insight if something s going wrong

Burkhard Hoeckendorf · Answer

Sure

Burkhard Hoeckendorf · Answer

Looks the same First encountered error is gt hasura 1 | internal could not connect to server Connection refused n tIs the server running on host postgres 172 31 0 2 and accepting n tTCP IP connections on port 5432 n path $ error connection error code postgres error

Burkhard Hoeckendorf · Answer

Might be a race condition This error appears before the first log output from Postgres

nicholas · Answer

Yeah that one isn t particularly concerning the various services will retry their connections as other containers start up hmm thinking face

Burkhard Hoeckendorf · Answer

I m in parallel trying another machine

nicholas · Answer

Smart I wasn t able to replicate this from a fresh pull myself

Burkhard Hoeckendorf · Answer

Looks like this is running fine Not sure what is going on here

Burkhard Hoeckendorf · Answer

Whatever it is seems to be a local issue though

nicholas · Answer

Meaning the second machine worked

Burkhard Hoeckendorf · Answer

Yes

Burkhard Hoeckendorf · Answer

Haven t run anything yet but dashboard shows no errors

nicholas · Answer

Oof sweat my only recommendation in that case would be to stop the server remove all the images with docker prune and then reinstall prefect in a new env and try again

Burkhard Hoeckendorf · Answer

Ok I ll try that If I was to stick to the other machine can I use the server remotely as in from the first machine

Burkhard Hoeckendorf · Answer

I meant stick with not stick to slightly smiling face

nicholas · Answer

Totally you ll need to set the `graphql url` server ui variable in the machine s config toml pointing it to the machine server is running from and restart the server before you ll be able to use the UI from your machine

nicholas · Answer

There are a bunch of other threads with info on that but if you re still having trouble after looking at those feel free to ping us

Burkhard Hoeckendorf · Answer

Ok Thanks a lot < nicholas> and < Zachary Hughes> Apologies that it ended up being a local issue

Zachary Hughes · Answer

All good These things happen Happy Prefect ing