f

    Frank Lind

    2 years ago
    I've tried various config file settings both in the .prefect/config.toml and by editing the python installed config in the site-packages directory. Had the same issue with the conda install. Any ideas?
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    Hey @Frank Lind, Can you share the traceback you're encountering? I wonder if you're facing the same issue as discussed in this thread.
    f

    Frank Lind

    2 years ago
    Hi, Thanks for the response. I didn't see an exact fix in the above thread. This is a pip install in a conda base environment :
    (base) av_developer@ubuntu:~$ prefect server start Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling scheduler ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... done Creating tmp_hasura_1 ... done Creating tmp_graphql_1 ... done Creating tmp_scheduler_1 ... done Creating tmp_apollo_1 ... done Creating tmp_ui_1 ... done Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_apollo_1, tmp_scheduler_1, tmp_ui_1 graphql_1 | graphql_1 | Running Alembic migrations... graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. hasura_1 | {"type":"startup","timestamp":"2020-06-22T20:25:17.297+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}} hasura_1 | {"type":"startup","timestamp":"2020-06-22T20:25:17.297+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgresql://prefect:...@postgres:5432/prefect_server"}}} hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host "postgres" (172.26.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"} apollo_1 | apollo_1 | > @ serve /apollo apollo_1 | > node dist/index.js apollo_1 | postgres_1 | The files belonging to this database system will be owned by user "postgres". postgres_1 | This user must also own the server process. postgres_1 | postgres_1 | The database cluster will be initialized with locale "en_US.utf8". postgres_1 | The default database encoding has accordingly been set to "UTF8". postgres_1 | The default text search configuration will be set to "english". postgres_1 | postgres_1 | Data page checksums are disabled. postgres_1 | postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok postgres_1 | creating subdirectories ... ok postgres_1 | selecting default max_connections ... 100 postgres_1 | selecting default shared_buffers ... 128MB postgres_1 | selecting default timezone ... Etc/UTC postgres_1 | selecting dynamic shared memory implementation ... posix postgres_1 | creating configuration files ... ok postgres_1 | running bootstrap script ... ok postgres_1 | performing post-bootstrap initialization ... ok graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 723da8052cbd, create extensions graphql_1 | Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_extension_name_index" graphql_1 | DETAIL: Key (extname)=(pgcrypto) already exists. graphql_1 | graphql_1 | [SQL: graphql_1 | CREATE EXTENSION IF NOT EXISTS "pgcrypto"; graphql_1 | CREATE EXTENSION IF NOT EXISTS "pg_trgm"; graphql_1 | ] graphql_1 | (Background on this error at: http://sqlalche.me/e/gkpj) graphql_1 | graphql_1 | Could not upgrade the database! postgres_1 | syncing data to disk ... postgres_1 | WARNING: enabling "trust" authentication for local connections postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or postgres_1 | --auth-local and --auth-host, the next time you run initdb. postgres_1 | ok postgres_1 | postgres_1 | Success. You can now start the database server using: postgres_1 | postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start postgres_1 | postgres_1 | waiting for server to start....2020-06-22 20:25:16.823 UTC [47] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-06-22 20:25:16.849 UTC [48] LOG: database system was shut down at 2020-06-22 20:25:16 UTC postgres_1 | 2020-06-22 20:25:16.857 UTC [47] LOG: database system is ready to accept connections postgres_1 | done postgres_1 | server started postgres_1 | CREATE DATABASE postgres_1 | postgres_1 | postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* postgres_1 | postgres_1 | 2020-06-22 20:25:17.366 UTC [47] LOG: received fast shutdown request postgres_1 | waiting for server to shut down....2020-06-22 20:25:17.367 UTC [47] LOG: aborting any active transactions postgres_1 | 2020-06-22 20:25:17.370 UTC [47] LOG: background worker "logical replication launcher" (PID 54) exited with exit code 1 postgres_1 | 2020-06-22 20:25:17.371 UTC [49] LOG: shutting down postgres_1 | 2020-06-22 20:25:17.382 UTC [47] LOG: database system is shut down postgres_1 | done postgres_1 | server stopped postgres_1 | postgres_1 | PostgreSQL init process complete; ready for start up. postgres_1 | postgres_1 | 2020-06-22 20:25:17.477 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres_1 | 2020-06-22 20:25:17.477 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres_1 | 2020-06-22 20:25:17.479 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-06-22 20:25:17.492 UTC [65] LOG: database system was shut down at 2020-06-22 20:25:17 UTC postgres_1 | 2020-06-22 20:25:17.495 UTC [1] LOG: database system is ready to accept connections postgres_1 | 2020-06-22 20:25:19.629 UTC [73] ERROR: duplicate key value violates unique constraint "pg_extension_name_index" postgres_1 | 2020-06-22 20:25:19.629 UTC [73] DETAIL: Key (extname)=(pgcrypto) already exists. postgres_1 | 2020-06-22 20:25:19.629 UTC [73] STATEMENT: postgres_1 | CREATE EXTENSION IF NOT EXISTS "pgcrypto"; postgres_1 | CREATE EXTENSION IF NOT EXISTS "pg_trgm"; postgres_1 | ui_1 | Replacing graphql references with: http://0.0.0.0:4200/graphql ui_1 | /var/www/js/app.7fd10635.js ui_1 | /var/www/js/chunk-290ef6aa.c9577213.js ui_1 | /var/www/js/chunk-35d72edc.aeb53d5b.js ui_1 | /var/www/js/chunk-5d657775.61e987a9.js ui_1 | /var/www/js/chunk-65b545d9.8422ea0b.js ui_1 | /var/www/js/chunk-6fd6fc38.7860b637.js ui_1 | /var/www/js/chunk-8fb1ff70.23e15619.js ui_1 | /var/www/js/chunk-93a4c748.ddd54c3b.js ui_1 | /var/www/js/chunk-c52f9fba.8c504a14.js ui_1 | /var/www/js/chunk-d03cee18.0c5fb4fe.js ui_1 | /var/www/js/chunk-d3b865a4.60462056.js ui_1 | /var/www/js/chunk-vendors.f1c338f7.js ui_1 | /var/www/js/dashboard.a5face6b.js ui_1 | /var/www/js/dashboardflow-version-groups.b35296c6.js ui_1 | /var/www/js/dashboardflowflow-runinteractive-apitasktask-run.64889ee7.js ui_1 | /var/www/js/dashboardflowflow-runschematics--viewtasktask-run.5dbabe8e.js ui_1 | /var/www/js/flow-run.e8d112a2.js ui_1 | /var/www/js/flow-version-groups.1891b372.js ui_1 | /var/www/js/flow.4d8eaec9.js ui_1 | /var/www/js/flowflow-runinteractive-apitasktask-run.12701769.js ui_1 | /var/www/js/flowflow-runschematics--viewtask~task-run.4d9b9eca.js ui_1 | /var/www/js/interactive-api.4362560d.js ui_1 | /var/www/js/onboard-page.4086ef69.js ui_1 | /var/www/js/schematics--view.38febea4.js ui_1 | /var/www/js/schematics.e3653fd6.js ui_1 | /var/www/js/task-run.d683a69e.js ui_1 | /var/www/js/task.5d12d448.js ui_1 | /var/www/js/welcome.df9a970c.js ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾 scheduler_1 | [2020-06-22 20:25:21,154] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 264 seconds... apollo_1 | 2020-06-22T20:25:21.889Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.26.0.4:4201 apollo_1 | 2020-06-22T20:25:21.890Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) apollo_1 | 2020-06-22T20:25:21.890Z apollo_1 | Trying again in 3 seconds... apollo_1 | apollo_1 | 2020-06-22T20:25:24.898Z Building schema... apollo_1 | 2020-06-22T20:25:25.947Z Building schema complete! apollo_1 | Server ready at http://0.0.0.0:4200 🚀 apollo_1 | 2020-06-22T20:25:26.076Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"5a9a3c85-cb0f-4f36-b5f2-f26fa264d3fd"}}
    I turned on the debug for the graphql in the config file...
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    Did you set the backend to server with
    prefect backend server
    before running
    prefect server start
    ? I ask because I am able to reproduce
    FetchError: request to <http://graphql:4201/health> failed, reason: connect ECONNREFUSED 172.26.0.4:4201
    while my backend is set to Cloud.
    f

    Frank Lind

    2 years ago
    I had tried it. Just repeated it and still get the same error trace.
    (base) av_developer@ubuntu:$ prefect backend server Backend switched to server (base) av_developer@ubuntu:$ prefect server start Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling scheduler ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... done Creating tmp_hasura_1 ... done Creating tmp_graphql_1 ... done Creating tmp_scheduler_1 ... done Creating tmp_apollo_1 ... done Creating tmp_ui_1 ... done
    ...
    ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾 apollo_1 | 2020-06-22T21:15:06.477Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.27.0.4:4201 apollo_1 | 2020-06-22T21:15:06.478Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7)
    Hi. I went to the developer instructions :
    and down into server/services/apollo and then did the npm install and start.
    npm run start
    @ start /home/av_developer/Downloads/prefect/server/services/apollo
    nodemon --exec babel-node src/index.js
    [nodemon] 1.19.4 [nodemon] to restart at any time, enter
    rs
    [nodemon] watching dir(s): src// ../../src// [nodemon] watching extensions: js,mjs,coffee,litcoffee,json,py [nodemon] starting
    babel-node src/index.js
    2020-06-22T21:29:00.255Z Error fetching GraphQL health: FetchError: request to http://localhost:4201/health failed, reason: connect ECONNREFUSED 127.0.0.1:4201 2020-06-22T21:29:00.266Z Error: Could not safely build a schema! at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js:126:11) at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema! at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js:126:11) at process._tickCallback (internal/process/next_tick.js:68:7) 2020-06-22T21:29:00.267Z Trying again in 3 seconds...
    This time without any conda environment.
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    Hey @Frank Lind - sorry to leave you hanging here. I'll have to come back with answers for you, as I have been encountering some weird behavior digging into this error. If you are able to submit a Github issue, it would be great to get more data on this, any insight to narrow down the problem is greatly appreciated.
    f

    Frank Lind

    2 years ago
    Hi Kyle, Graphql Apollo service failure under Ubuntu 20.04 LTS #2842, now in Github.
    Kyle Moon-Wright

    Kyle Moon-Wright

    2 years ago
    Fantastic, thank you! I'll be watching that one closely.