Thread
#prefect-server
    frojo

    frojo

    7 months ago
    Hi everyone! I'm new to Prefect (just a couple of months). I've been using it as provided for some local tests without an issue. I was trying to replicate this tests on a remote machine (ssh mediated) but I have runned into some issues. I'm trying to spin up a local prefect server (just testing some flows) on a remote machine. Some data on versions prefect diagnostics:
    {
    "config_overrides": {},
    "env_vars": [],
    "system_information": {
    "platform": "Linux-5.4.0-96-generic-x86_64-with-glibc2.27",
    "prefect_backend": "server",
    "prefect_version": "0.15.13",
    "python_version": "3.8.12"
    }
    }
    I was setting the local server as option and started as usual:
    prefect backend server
    prefect server start
    No special options. Images are pulled and all the services started. However some errors (regarding graphql mainly) start to apper on logs.
    graphql_1   | Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
    graphql_1   | 
    graphql_1   | Could not upgrade the database!
    graphql_1   | Error: Failed to apply Hasura metadata! Error: 503 Server Error: Service Unavailable for url: <http://hasura:3000/v1/query>
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    tmp_graphql_1 exited with code 1
    (I have the full trace until stopping them). I also checked (while waiting the spin up - just displaying the failing one)
    > docker ps 
    7036b92f538d   prefecthq/server:core-0.15.13   "tini -g -- bash -c …"   About a minute ago   Restarting (1) 1 second ago                                      tmp_graphql_
    which is continously restarting. I have tryied: • previuos Prefect version (0.15.0) with the same effect. • the same env on my local machine with success (so I suspect from the remote configuration, but I could not figure what is missing) Thanks!
    Anna Geller

    Anna Geller

    7 months ago
    Could you move the code blocks e.g. docker ps output to the thread? This helps us keep the main channel cleaner. I think your issue might be related to the --expose flag. You would need to start the server using:
    prefect server start --expose
    Regarding the GraphQL errors with tini, it may be related to this Github issue Since you are still new to Prefect, perhaps you can use Prefect Cloud at first? You can sign up and use Prefect Cloud completely free and run up to 20,000 task runs each month (no credit card required). Once you set up your workflows, and became familiar even more with Prefect, you can always change to Server later - your flows will work the same way. And until then, the error you see will likely be fixed.
    frojo

    frojo

    7 months ago
    Hi @Anna Geller! Thanks for your answer. I will give it a try ( --expose flag). Regarding the GraphQL errors I'm reading the mentioned issue. With respecto to Prefect Cloud, due to some security issues the machine is only allowed to talk to our inner net (no internet conection). Regarding to the channel if you believe I have to edit the post (to shorten it) just tell me 👍 (I've already shrink the docker ps part 😄)
    Michael Adkins

    Michael Adkins

    7 months ago
    Hey @frojo — is the Hasura container outputting any logs?
    “Service unavailable” seems to indicate Hasura isn’t starting correctly
    frojo

    frojo

    7 months ago
    Hi @Michael Adkins! I was trying @Anna Geller suggestion (--expose flag) with no success. I will now check Hasura docker logs. Thanks!
    Hasura docker logs:
    {"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
    {"type":"pg-client","timestamp":"2022-02-02T15:57:34.406+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
    {"type":"startup","timestamp":"2022-02-02T15:57:34.406+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
    {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.25.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
    {"type":"unstructured","timestamp":"2022-02-02T15:57:40.289+0000","level":"warn","detail":"The following environment variables are deprecated: HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE"}
    It seems that hasura is having some issues with postgres. However postgres service seem to be up and running (at least from docker ps side). I will check now postgres logs
    Michael Adkins

    Michael Adkins

    7 months ago
    🙂 Looks like postgres isn’t running
    Haha
    Yeah.. gotta follow the chain with docker-compose
    frojo

    frojo

    7 months ago
    postgres logs:
    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    
    The database cluster will be initialized with locale "en_US.utf8".
    The default database encoding has accordingly been set to "UTF8".
    The default text search configuration will be set to "english".
    
    Data page checksums are disabled.
    
    fixing permissions on existing directory /var/lib/postgresql/data ... ok
    creating subdirectories ... ok
    selecting default max_connections ... 100
    selecting default shared_buffers ... 128MB
    selecting default timezone ... Etc/UTC
    selecting dynamic shared memory implementation ... posix
    creating configuration files ... ok
    running bootstrap script ... ok
    performing post-bootstrap initialization ... ok
    
    WARNING: enabling "trust" authentication for local connections
    You can change this by editing pg_hba.conf or using the option -A, or
    --auth-local and --auth-host, the next time you run initdb.
    syncing data to disk ... ok
    
    Success. You can now start the database server using:
    
        pg_ctl -D /var/lib/postgresql/data -l logfile start
    
    waiting for server to start....2022-02-02 15:57:33.978 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
    2022-02-02 15:57:34.175 UTC [49] LOG:  database system was shut down at 2022-02-02 15:57:31 UTC
    2022-02-02 15:57:34.370 UTC [48] LOG:  database system is ready to accept connections
     done
    server started
    CREATE DATABASE
    
    
    /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
    
    2022-02-02 15:57:36.915 UTC [48] LOG:  received fast shutdown request
    waiting for server to shut down....2022-02-02 15:57:37.029 UTC [48] LOG:  aborting any active transactions
    2022-02-02 15:57:37.031 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
    2022-02-02 15:57:37.033 UTC [50] LOG:  shutting down
    2022-02-02 15:57:37.494 UTC [48] LOG:  database system is shut down
     done
    server stopped
    
    PostgreSQL init process complete; ready for start up.
    
    2022-02-02 15:57:37.573 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
    2022-02-02 15:57:37.573 UTC [1] LOG:  listening on IPv6 address "::", port 5432
    2022-02-02 15:57:37.699 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
    2022-02-02 15:57:37.887 UTC [76] LOG:  database system was shut down at 2022-02-02 15:57:37 UTC
    2022-02-02 15:57:37.937 UTC [1] LOG:  database system is ready to accept connections
    2022-02-02 15:57:40.250 UTC [84] ERROR:  duplicate key value violates unique constraint "pg_extension_name_index"
    2022-02-02 15:57:40.250 UTC [84] DETAIL:  Key (extname)=(pgcrypto) already exists.
    2022-02-02 15:57:40.250 UTC [84] STATEMENT:
                    CREATE EXTENSION IF NOT EXISTS "pgcrypto";
                    CREATE EXTENSION IF NOT EXISTS "pg_trgm";
                    SET TIME ZONE 'UTC';
    Thanks @Michael Adkins I will be digging into it until I find it or it beats me 🤣
    Michael Adkins

    Michael Adkins

    7 months ago
    Hmm…
    2022-02-02 15:57:40.250 UTC [84] ERROR: duplicate key value violates unique constraint “pg_extension_name_index”
    tends to happen when two database upgrade requests run simultaneously
    Which isn’t particularly helpful 🙂 What happens if you start with
    --no-upgrade
    ?
    If it starts happily then, you can run migrations manually
    ❯ docker exec t-graphql-1 prefect-server database upgrade -y
    frojo

    frojo

    7 months ago
    --no-upgrade
    made the trick for hasura docker (it is now running). But still reports that its unable to conect to the postgres port
    {
      "internal": "could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.19.0.2) and accepting\n\tTCP/IP connections on port 5432?\n",
      "path": "$",
      "error": "connection error",
      "code": "postgres-error"
    }
    While postgres is running and with log
    PostgreSQL init process complete; ready for start up.
    
    2022-02-02 17:49:09.270 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
    2022-02-02 17:49:09.270 UTC [1] LOG:  listening on IPv6 address "::", port 5432
    2022-02-02 17:49:09.347 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
    2022-02-02 17:49:09.486 UTC [77] LOG:  database system was shut down at 2022-02-02 17:49:09 UTC
    2022-02-02 17:49:09.563 UTC [1] LOG:  database system is ready to accept connections
    Thanks again @Michael Adkins, i will now check step by step (the docker compose up process) and will come back if something new araises. 👍