Struggling after Update to 0.14.12 to start prefect-server. Hi all 👋 . I just upgraded to prefect 0.14.12 (from 0.14.4) running server on a machine of us (hosting more docker apps, so maybe something changed outside of my control as other people use it to start apps, too). I do connect to the system via VPN which might create more complications? After the update my command:

prefect server start --use-volume -d

is pulling the images, creating the network and creating the parts (tmp_postgres_1, tmp_hasura_1, …). However, it never finishes (no detach) and I do get errors in the logs of tmp_apollo_1:

Trying again in 3 seconds...
Building schema...
FetchError: request to <http://hasura:3001/v1alpha1/graphql> failed, reason: connect ECONNREFUSED 192.168.64.3:3001
    at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:315:20)
    at Socket.socketErrorListener (_http_client.js:469:9)
    at Socket.emit (events.js:315:20)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}
Trying again in 3 seconds...

Not sure what’s going on? My config is pretty simple, too:

[server]
    [server.database]
    host = "localhost"
    port = "5432"
    host_port = "5434"

    [server.hasura]
    host = "localhost"
    port = "3000"
    host_port = "3001"

The micro-services seem to be up and running:

(prefect) [root@myserver:~]$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                    PORTS                              NAMES
aec1a48767a3        prefecthq/ui:core-0.14.12       "/docker-entrypoint.…"   17 minutes ago      Up 17 minutes (healthy)   80/tcp, 0.0.0.0:8080->8080/tcp     tmp_ui_1
2996c1840fe1        prefecthq/apollo:core-0.14.12   "tini -g -- bash -c …"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:4200->4200/tcp             tmp_apollo_1
2263ca7701e6        prefecthq/server:core-0.14.12   "tini -g -- python s…"   17 minutes ago      Up 17 minutes                                                tmp_towel_1
ed29fdb1b3b0        prefecthq/server:core-0.14.12   "tini -g -- bash -c …"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:4201->4201/tcp             tmp_graphql_1
2a3dffb7d641        hasura/graphql-engine:v1.3.3    "graphql-engine serve"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:3001->3000/tcp             tmp_hasura_1
cafbde90dc48        postgres:11                     "docker-entrypoint.s…"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:5434->5432/tcp             tmp_postgres_1

Hi @Christian - by default Prefect Server doesn't detach from the terminal - you'll need to use something like

screen

or

nohup

for that; are you able to query your endpoint through the browser? Navigating to

<http://localhost:4200/graphql>

should allow you to visit the graphql playground that comes with the server

Hi. I cannot reach this endpoint. After digging through the forums I also changed me setup to this config:

[server]
    [server.database]
    port = "5432"
    host_port = "5434"

    [server.hasura]
    host = "<http://172.27.80.xxx>"
    port = "3001"
    host_port = "3001"
    graphql_url = "http://${server.hasura.host}:${server.hasura.port}/v1alpha1/graphql"
    ws_url = "ws://${server.hasura.host}:${server.hasura.port}/v1alpha1/graphql"
    execute_retry_seconds = 10

    [server.graphql]
    host = "<http://172.27.80.xxx>"
    port = "4201"
    host_port = "4201"
    debug = false
    path = "/graphql"
  
    [server.ui]
    apollo_url = "<http://172.27.80.xxx:4200/graphql>"

I currently see this in the logs when I try to bring up the system:

apollo_1    | Building schema...                                                                                                                                                                      
apollo_1    | FetchError: request to <http://hasura:3001/v1alpha1/graphql> failed, reason: connect ECONNREFUSED 192.168.144.3:3001                                                                      
apollo_1    |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1461:11)                                                                                                 
apollo_1    |     at ClientRequest.emit (events.js:315:20)                                                                                                                                            
apollo_1    |     at Socket.socketErrorListener (_http_client.js:469:9)                                                                                                                               
apollo_1    |     at Socket.emit (events.js:315:20)                                                                                                                                                   
apollo_1    |     at emitErrorNT (internal/streams/destroy.js:106:8)                                                                                                                                  
apollo_1    |     at emitErrorCloseNT (internal/streams/destroy.js:74:3)
apollo_1    |     at processTicksAndRejections (internal/process/task_queues.js:80:21) {
apollo_1    |   type: 'system',
apollo_1    |   errno: 'ECONNREFUSED',
apollo_1    |   code: 'ECONNREFUSED'
apollo_1    | }

That makes sense if the service isn't able to start; can you confirm a few things? 1. Do you have anything else running on any of those ports that might interfere with service spinup? 2. Since you're upgrading your version of Server, can you confirm that the database migrations ran?

4200 should be available. The migration ran as far as I can tell … is there any way to verify this? Also, I’m a bit puzzled by the url reported in the errors…

request to <http://hasura:3001/v1alpha1/graphql> failed

Is this a valid URL?

That URL seems fine - you'll see

alembic

migrations run on Server startup. This looks odd to me, is there a reason these ports are different in your config?

[server.database]
    port = "5432"
    host_port = "5434"

I have port 5432 taken by another container… After tinkering some more in my config I now see this when I start up:

Starting tmp_hasura_1 ... done

ERROR: for graphql  Container "fab6c18bd8fc" is unhealthy.
ERROR: Encountered errors while bringing up the project.

Should I

docker system prune -a

?

That couldn't hurt but what sort of resources are you allocating to Docker? We've seen instances of this when Docker doesn't have enough resources

Should be plenty I think (I’m not the admin on the box)… MEM LIMIT is 503.8GiB I reverted my tinkering an now the stack comes up but never completes. Constant recurrence of the above error message. Also, cannot reach graphql via the browser… bummer

This sounds like you're running into a combination of configuration issues and this one: https://github.com/PrefectHQ/prefect/issues/4170 - if nothing in that thread is helpful, would you mind adding some details about your system info to that ticket?

Sure. Will do. I’ll switch to cloud for the moment but would be good to get a solution to this since we want to use an on-prem install . Thanks