frojo
02/02/2022, 12:11 PM{
"config_overrides": {},
"env_vars": [],
"system_information": {
"platform": "Linux-5.4.0-96-generic-x86_64-with-glibc2.27",
"prefect_backend": "server",
"prefect_version": "0.15.13",
"python_version": "3.8.12"
}
}
I was setting the local server as option and started as usual:
prefect backend server
prefect server start
No special options. Images are pulled and all the services started. However some errors (regarding graphql mainly) start to apper on logs.
graphql_1 | Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1 |
graphql_1 | Could not upgrade the database!
graphql_1 | Error: Failed to apply Hasura metadata! Error: 503 Server Error: Service Unavailable for url: <http://hasura:3000/v1/query>
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
tmp_graphql_1 exited with code 1
(I have the full trace until stopping them).
I also checked (while waiting the spin up - just displaying the failing one)
> docker ps
7036b92f538d prefecthq/server:core-0.15.13 "tini -g -- bash -c …" About a minute ago Restarting (1) 1 second ago tmp_graphql_
which is continously restarting.
I have tryied:
• previuos Prefect version (0.15.0) with the same effect.
• the same env on my local machine with success (so I suspect from the remote configuration, but I could not figure what is missing)
Thanks!Anna Geller
02/02/2022, 2:03 PMprefect server start --expose
Regarding the GraphQL errors with tini, it may be related to this Github issue
Since you are still new to Prefect, perhaps you can use Prefect Cloud at first? You can sign up and use Prefect Cloud completely free and run up to 20,000 task runs each month (no credit card required). Once you set up your workflows, and became familiar even more with Prefect, you can always change to Server later - your flows will work the same way. And until then, the error you see will likely be fixed.frojo
02/02/2022, 2:19 PMZanie
02/02/2022, 2:57 PMfrojo
02/02/2022, 3:02 PM{"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
{"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
{"type":"startup","timestamp":"2022-02-02T15:57:34.406+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
{"type":"unstructured","timestamp":"2022-02-02T15:57:40.289+0000","level":"warn","detail":"The following environment variables are deprecated: HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE"}
Zanie
02/02/2022, 3:15 PMfrojo
02/02/2022, 3:16 PMThe files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2022-02-02 15:57:33.978 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 15:57:34.175 UTC [49] LOG: database system was shut down at 2022-02-02 15:57:31 UTC
2022-02-02 15:57:34.370 UTC [48] LOG: database system is ready to accept connections
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2022-02-02 15:57:36.915 UTC [48] LOG: received fast shutdown request
waiting for server to shut down....2022-02-02 15:57:37.029 UTC [48] LOG: aborting any active transactions
2022-02-02 15:57:37.031 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
2022-02-02 15:57:37.033 UTC [50] LOG: shutting down
2022-02-02 15:57:37.494 UTC [48] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2022-02-02 15:57:37.573 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-02-02 15:57:37.573 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-02-02 15:57:37.699 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 15:57:37.887 UTC [76] LOG: database system was shut down at 2022-02-02 15:57:37 UTC
2022-02-02 15:57:37.937 UTC [1] LOG: database system is ready to accept connections
2022-02-02 15:57:40.250 UTC [84] ERROR: duplicate key value violates unique constraint "pg_extension_name_index"
2022-02-02 15:57:40.250 UTC [84] DETAIL: Key (extname)=(pgcrypto) already exists.
2022-02-02 15:57:40.250 UTC [84] STATEMENT:
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";
SET TIME ZONE 'UTC';
Thanks @Zanie I will be digging into it until I find it or it beats me 🤣Zanie
02/02/2022, 3:18 PM2022-02-02 15:57:40.250 UTC [84] ERROR: duplicate key value violates unique constraint “pg_extension_name_index”tends to happen when two database upgrade requests run simultaneously
--no-upgrade
?❯ docker exec t-graphql-1 prefect-server database upgrade -y
frojo
02/02/2022, 5:07 PM--no-upgrade
made the trick for hasura docker (it is now running). But still reports that its unable to conect to the postgres port
{
"internal": "could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.19.0.2) and accepting\n\tTCP/IP connections on port 5432?\n",
"path": "$",
"error": "connection error",
"code": "postgres-error"
}
While postgres is running and with log
PostgreSQL init process complete; ready for start up.
2022-02-02 17:49:09.270 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-02-02 17:49:09.270 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-02-02 17:49:09.347 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 17:49:09.486 UTC [77] LOG: database system was shut down at 2022-02-02 17:49:09 UTC
2022-02-02 17:49:09.563 UTC [1] LOG: database system is ready to accept connections
Thanks again @Zanie, i will now check step by step (the docker compose up process) and will come back if something new araises. 👍