I've tried various config file settings both in th...
# prefect-community
f
I've tried various config file settings both in the .prefect/config.toml and by editing the python installed config in the site-packages directory. Had the same issue with the conda install. Any ideas?
k
Hey @Frank Lind, Can you share the traceback you're encountering? I wonder if you're facing the same issue as discussed in this thread.
f
Hi, Thanks for the response. I didn't see an exact fix in the above thread. This is a pip install in a conda base environment :
(base) av_developer@ubuntu:~$ prefect server start Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling scheduler ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... done Creating tmp_hasura_1 ... done Creating tmp_graphql_1 ... done Creating tmp_scheduler_1 ... done Creating tmp_apollo_1 ... done Creating tmp_ui_1 ... done Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_apollo_1, tmp_scheduler_1, tmp_ui_1 graphql_1 | graphql_1 | Running Alembic migrations... graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. hasura_1 | {"type":"startup","timestamp":"2020-06-22T202517.297+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}} hasura_1 | {"type":"startup","timestamp":"2020-06-22T202517.297+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgresql://prefect:...@postgres:5432/prefect_server"}}} hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.26.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"} apollo_1 | apollo_1 | > @ serve /apollo apollo_1 | > node dist/index.js apollo_1 | postgres_1 | The files belonging to this database system will be owned by user "postgres". postgres_1 | This user must also own the server process. postgres_1 | postgres_1 | The database cluster will be initialized with locale "en_US.utf8". postgres_1 | The default database encoding has accordingly been set to "UTF8". postgres_1 | The default text search configuration will be set to "english". postgres_1 | postgres_1 | Data page checksums are disabled. postgres_1 | postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok postgres_1 | creating subdirectories ... ok postgres_1 | selecting default max_connections ... 100 postgres_1 | selecting default shared_buffers ... 128MB postgres_1 | selecting default timezone ... Etc/UTC postgres_1 | selecting dynamic shared memory implementation ... posix postgres_1 | creating configuration files ... ok postgres_1 | running bootstrap script ... ok postgres_1 | performing post-bootstrap initialization ... ok graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 723da8052cbd, create extensions graphql_1 | Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_extension_name_index" graphql_1 | DETAIL: Key (extname)=(pgcrypto) already exists. graphql_1 | graphql_1 | [SQL: graphql_1 | CREATE EXTENSION IF NOT EXISTS "pgcrypto"; graphql_1 | CREATE EXTENSION IF NOT EXISTS "pg_trgm"; graphql_1 | ] graphql_1 | (Background on this error at: http://sqlalche.me/e/gkpj) graphql_1 | graphql_1 | Could not upgrade the database! postgres_1 | syncing data to disk ... postgres_1 | WARNING: enabling "trust" authentication for local connections postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or postgres_1 | --auth-local and --auth-host, the next time you run initdb. postgres_1 | ok postgres_1 | postgres_1 | Success. You can now start the database server using: postgres_1 | postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start postgres_1 | postgres_1 | waiting for server to start....2020-06-22 202516.823 UTC [47] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-06-22 202516.849 UTC [48] LOG: database system was shut down at 2020-06-22 202516 UTC postgres_1 | 2020-06-22 202516.857 UTC [47] LOG: database system is ready to accept connections postgres_1 | done postgres_1 | server started postgres_1 | CREATE DATABASE postgres_1 | postgres_1 | postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* postgres_1 | postgres_1 | 2020-06-22 202517.366 UTC [47] LOG: received fast shutdown request postgres_1 | waiting for server to shut down....2020-06-22 202517.367 UTC [47] LOG: aborting any active transactions postgres_1 | 2020-06-22 202517.370 UTC [47] LOG: background worker "logical replication launcher" (PID 54) exited with exit code 1 postgres_1 | 2020-06-22 202517.371 UTC [49] LOG: shutting down postgres_1 | 2020-06-22 202517.382 UTC [47] LOG: database system is shut down postgres_1 | done postgres_1 | server stopped postgres_1 | postgres_1 | PostgreSQL init process complete; ready for start up. postgres_1 | postgres_1 | 2020-06-22 202517.477 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres_1 | 2020-06-22 202517.477 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres_1 | 2020-06-22 202517.479 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-06-22 202517.492 UTC [65] LOG: database system was shut down at 2020-06-22 202517 UTC postgres_1 | 2020-06-22 202517.495 UTC [1] LOG: database system is ready to accept connections postgres_1 | 2020-06-22 202519.629 UTC [73] ERROR: duplicate key value violates unique constraint "pg_extension_name_index" postgres_1 | 2020-06-22 202519.629 UTC [73] DETAIL: Key (extname)=(pgcrypto) already exists. postgres_1 | 2020-06-22 202519.629 UTC [73] STATEMENT: postgres_1 | CREATE EXTENSION IF NOT EXISTS "pgcrypto"; postgres_1 | CREATE EXTENSION IF NOT EXISTS "pg_trgm"; postgres_1 | ui_1 | Replacing graphql references with: http://0.0.0.0:4200/graphql ui_1 | /var/www/js/app.7fd10635.js ui_1 | /var/www/js/chunk-290ef6aa.c9577213.js ui_1 | /var/www/js/chunk-35d72edc.aeb53d5b.js ui_1 | /var/www/js/chunk-5d657775.61e987a9.js ui_1 | /var/www/js/chunk-65b545d9.8422ea0b.js ui_1 | /var/www/js/chunk-6fd6fc38.7860b637.js ui_1 | /var/www/js/chunk-8fb1ff70.23e15619.js ui_1 | /var/www/js/chunk-93a4c748.ddd54c3b.js ui_1 | /var/www/js/chunk-c52f9fba.8c504a14.js ui_1 | /var/www/js/chunk-d03cee18.0c5fb4fe.js ui_1 | /var/www/js/chunk-d3b865a4.60462056.js ui_1 | /var/www/js/chunk-vendors.f1c338f7.js ui_1 | /var/www/js/dashboard.a5face6b.js ui_1 | /var/www/js/dashboard~flow-version-groups.b35296c6.js ui_1 | /var/www/js/dashboard~flow~flow-run~interactive-api~task~task-run.64889ee7.js ui_1 | /var/www/js/dashboard~flow~flow-run~schematics--view~task~task-run.5dbabe8e.js ui_1 | /var/www/js/flow-run.e8d112a2.js ui_1 | /var/www/js/flow-version-groups.1891b372.js ui_1 | /var/www/js/flow.4d8eaec9.js ui_1 | /var/www/js/flow~flow-run~interactive-api~task~task-run.12701769.js ui_1 | /var/www/js/flow~flow-run~schematics--view~task~task-run.4d9b9eca.js ui_1 | /var/www/js/interactive-api.4362560d.js ui_1 | /var/www/js/onboard-page.4086ef69.js ui_1 | /var/www/js/schematics--view.38febea4.js ui_1 | /var/www/js/schematics.e3653fd6.js ui_1 | /var/www/js/task-run.d683a69e.js ui_1 | /var/www/js/task.5d12d448.js ui_1 | /var/www/js/welcome.df9a970c.js ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾 scheduler_1 | [2020-06-22 202521,154] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 264 seconds... apollo_1 | 2020-06-22T202521.889Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.26.0.4:4201 apollo_1 | 2020-06-22T202521.890Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js12911) apollo_1 | at process._tickCallback (internal/process/next_tick.js687) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js12911) apollo_1 | at process._tickCallback (internal/process/next_tick.js687) apollo_1 | 2020-06-22T202521.890Z apollo_1 | Trying again in 3 seconds... apollo_1 | apollo_1 | 2020-06-22T202524.898Z Building schema... apollo_1 | 2020-06-22T202525.947Z Building schema complete! apollo_1 | Server ready at http://0.0.0.0:4200 🚀 apollo_1 | 2020-06-22T202526.076Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"5a9a3c85-cb0f-4f36-b5f2-f26fa264d3fd"}}
I turned on the debug for the graphql in the config file...
k
Did you set the backend to server with
prefect backend server
before running
prefect server start
? I ask because I am able to reproduce
FetchError: request to <http://graphql:4201/health> failed, reason: connect ECONNREFUSED 172.26.0.4:4201
while my backend is set to Cloud.
f
I had tried it. Just repeated it and still get the same error trace.
(base) av_developer@ubuntu:~$ prefect backend server Backend switched to server (base) av_developer@ubuntu:~$ prefect server start Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling scheduler ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... done Creating tmp_hasura_1 ... done Creating tmp_graphql_1 ... done Creating tmp_scheduler_1 ... done Creating tmp_apollo_1 ... done Creating tmp_ui_1 ... done
...
ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾 apollo_1 | 2020-06-22T211506.477Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.27.0.4:4201 apollo_1 | 2020-06-22T211506.478Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js12911) apollo_1 | at process._tickCallback (internal/process/next_tick.js687) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js12911) apollo_1 | at process._tickCallback (internal/process/next_tick.js687)
Hi. I went to the developer instructions :
and down into server/services/apollo and then did the npm install and start.
npm run start
@ start /home/av_developer/Downloads/prefect/server/services/apollo
nodemon --exec babel-node src/index.js
[nodemon] 1.19.4 [nodemon] to restart at any time, enter
rs
[nodemon] watching dir(s): src/**/* ../../src/**/* [nodemon] watching extensions: js,mjs,coffee,litcoffee,json,py [nodemon] starting
babel-node src/index.js
2020-06-22T212900.255Z Error fetching GraphQL health: FetchError: request to http://localhost:4201/health failed, reason: connect ECONNREFUSED 127.0.0.1:4201 2020-06-22T212900.266Z Error: Could not safely build a schema! at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js12611) at process._tickCallback (internal/process/next_tick.js687) Could not safely build a schema! Error: Could not safely build a schema! at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js12611) at process._tickCallback (internal/process/next_tick.js687) 2020-06-22T212900.267Z Trying again in 3 seconds...
This time without any conda environment.
k
Hey @Frank Lind - sorry to leave you hanging here. I'll have to come back with answers for you, as I have been encountering some weird behavior digging into this error. If you are able to submit a Github issue, it would be great to get more data on this, any insight to narrow down the problem is greatly appreciated.
f
Hi Kyle, Graphql Apollo service failure under Ubuntu 20.04 LTS #2842, now in Github.
🎉 1
k
Fantastic, thank you! I'll be watching that one closely.