https://prefect.io logo
Join Slack
Powered by
Hi everyone! I'm new to Prefect (just a couple of ...
# prefect-server
f
Hi everyone! I'm new to Prefect (just a couple of months). I've been using it as provided for some local tests without an issue. I was trying to replicate this tests on a remote machine (ssh mediated) but I have runned into some issues. I'm trying to spin up a local prefect server (just testing some flows) on a remote machine. Some data on versions `prefect diagnostics`: 
{
"config_overrides": {},
"env_vars": [],
"system_information": {
"platform": "Linux-5.4.0-96-generic-x86_64-with-glibc2.27",
"prefect_backend": "server",
"prefect_version": "0.15.13",
"python_version": "3.8.12"
}
}
I was setting the local server as option and started as usual:
Copy code
prefect backend server
prefect server start
No special options. Images are pulled and all the services started. However some errors (regarding graphql mainly) start to apper on logs.
Copy code
graphql_1   | Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1   | 
graphql_1   | Could not upgrade the database!
graphql_1   | Error: Failed to apply Hasura metadata! Error: 503 Server Error: Service Unavailable for url: <http://hasura:3000/v1/query>
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
tmp_graphql_1 exited with code 1
(I have the full trace until stopping them). I also checked (while waiting the spin up - just displaying the failing one)
Copy code
> docker ps 
7036b92f538d   prefecthq/server:core-0.15.13   "tini -g -- bash -c …"   About a minute ago   Restarting (1) 1 second ago                                      tmp_graphql_
which is continously restarting. I have tryied: • previuos Prefect version (0.15.0) with the same effect. • the same env on my local machine with success (so I suspect from the remote configuration, but I could not figure what is missing) Thanks!
a
Could you move the code blocks e.g. docker ps output to the thread? This helps us keep the main channel cleaner. I think your issue might be related to the --expose flag. You would need to start the server using:
Copy code
prefect server start --expose
Regarding the GraphQL errors with tini, it may be related to this Github issue Since you are still new to Prefect, perhaps you can use Prefect Cloud at first? You can sign up and use Prefect Cloud completely free and run up to 20,000 task runs each month (no credit card required). Once you set up your workflows, and became familiar even more with Prefect, you can always change to Server later - your flows will work the same way. And until then, the error you see will likely be fixed.
f
Hi @Anna Geller! Thanks for your answer. I will give it a try ( --expose flag). Regarding the GraphQL errors I'm reading the mentioned issue. With respecto to Prefect Cloud, due to some security issues the machine is only allowed to talk to our inner net (no internet conection). Regarding to the channel if you believe I have to edit the post (to shorten it) just tell me 👍 (I've already shrink the docker ps part 😄)
z
Hey @frojo — is the Hasura container outputting any logs?
“Service unavailable” seems to indicate Hasura isn’t starting correctly
upvote 1
f
Hi @Zanie! I was trying @Anna Geller suggestion (--expose flag) with no success. I will now check Hasura docker logs. Thanks!
👍 1
Hasura docker logs:
Copy code
{"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
{"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
{"type":"startup","timestamp":"2022-02-02T15:57:34.406+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
{"type":"unstructured","timestamp":"2022-02-02T15:57:40.289+0000","level":"warn","detail":"The following environment variables are deprecated: HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE"}
It seems that hasura is having some issues with postgres. However postgres service seem to be up and running (at least from docker ps side). I will check now postgres logs
z
🙂 Looks like postgres isn’t running
Haha
Yeah.. gotta follow the chain with docker-compose
f
postgres logs:
Copy code
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2022-02-02 15:57:33.978 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 15:57:34.175 UTC [49] LOG:  database system was shut down at 2022-02-02 15:57:31 UTC
2022-02-02 15:57:34.370 UTC [48] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2022-02-02 15:57:36.915 UTC [48] LOG:  received fast shutdown request
waiting for server to shut down....2022-02-02 15:57:37.029 UTC [48] LOG:  aborting any active transactions
2022-02-02 15:57:37.031 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
2022-02-02 15:57:37.033 UTC [50] LOG:  shutting down
2022-02-02 15:57:37.494 UTC [48] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-02-02 15:57:37.573 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-02 15:57:37.573 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-02-02 15:57:37.699 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 15:57:37.887 UTC [76] LOG:  database system was shut down at 2022-02-02 15:57:37 UTC
2022-02-02 15:57:37.937 UTC [1] LOG:  database system is ready to accept connections
2022-02-02 15:57:40.250 UTC [84] ERROR:  duplicate key value violates unique constraint "pg_extension_name_index"
2022-02-02 15:57:40.250 UTC [84] DETAIL:  Key (extname)=(pgcrypto) already exists.
2022-02-02 15:57:40.250 UTC [84] STATEMENT:
                CREATE EXTENSION IF NOT EXISTS "pgcrypto";
                CREATE EXTENSION IF NOT EXISTS "pg_trgm";
                SET TIME ZONE 'UTC';
Thanks @Zanie I will be digging into it until I find it or it beats me 🤣
z
Hmm…
2022-02-02 155740.250 UTC [84] ERROR: duplicate key value violates unique constraint “pg_extension_name_index”
tends to happen when two database upgrade requests run simultaneously
👍 1
Which isn’t particularly helpful 🙂 What happens if you start with 
--no-upgrade
?
If it starts happily then, you can run migrations manually
Copy code
❯ docker exec t-graphql-1 prefect-server database upgrade -y
f
--no-upgrade
made the trick for hasura docker (it is now running). But still reports that its unable to conect to the postgres port
Copy code
{
  "internal": "could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.19.0.2) and accepting\n\tTCP/IP connections on port 5432?\n",
  "path": "$",
  "error": "connection error",
  "code": "postgres-error"
}
While postgres is running and with log
Copy code
PostgreSQL init process complete; ready for start up.

2022-02-02 17:49:09.270 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-02 17:49:09.270 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-02-02 17:49:09.347 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-02 17:49:09.486 UTC [77] LOG:  database system was shut down at 2022-02-02 17:49:09 UTC
2022-02-02 17:49:09.563 UTC [1] LOG:  database system is ready to accept connections
Thanks again @Zanie, i will now check step by step (the docker compose up process) and will come back if something new araises. 👍
2 Views
Open in Slack
PreviousNext
Powered by