https://prefect.io logo
Join Slack
Powered by
Hi all! I'm having a bit of an issue with an empty...
# ask-community
p
Hi all! I'm having a bit of an issue with an empty UI when running 
prefect server start
. I'll post the logs in the thread.
Copy code
Attaching to t_postgres_1, t_hasura_1, t_graphql_1, t_towel_1, t_apollo_1, t_ui_1
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
hasura_1    | {"type":"pg-client","timestamp":"2021-01-27T22:21:57.028+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
postgres_1  | The files belonging to this database system will be owned by user "postgres".
postgres_1  | This user must also own the server process.
postgres_1  | 
postgres_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres_1  | The default database encoding has accordingly been set to "UTF8".
postgres_1  | The default text search configuration will be set to "english".
postgres_1  | 
postgres_1  | Data page checksums are disabled.
postgres_1  | 
postgres_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1  | creating subdirectories ... ok
postgres_1  | selecting default max_connections ... 100
postgres_1  | selecting default shared_buffers ... 128MB
postgres_1  | selecting default timezone ... Etc/UTC
postgres_1  | selecting dynamic shared memory implementation ... posix
postgres_1  | creating configuration files ... ok
postgres_1  | running bootstrap script ... ok
postgres_1  | performing post-bootstrap initialization ... ok
postgres_1  | syncing data to disk ... ok
postgres_1  | 
postgres_1  | Success. You can now start the database server using:
postgres_1  | 
postgres_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1  | 
postgres_1  | 
postgres_1  | WARNING: enabling "trust" authentication for local connections
postgres_1  | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1  | --auth-local and --auth-host, the next time you run initdb.
postgres_1  | waiting for server to start....2021-01-27 22:21:57.186 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2021-01-27 22:21:57.199 UTC [49] LOG:  database system was shut down at 2021-01-27 22:21:56 UTC
postgres_1  | 2021-01-27 22:21:57.209 UTC [48] LOG:  database system is ready to accept connections
hasura_1    | {"type":"pg-client","timestamp":"2021-01-27T22:21:57.028+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1    | {"type":"startup","timestamp":"2021-01-27T22:21:57.028+0000","level":"error","detail":{"kind":"db_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
postgres_1  |  done
postgres_1  | server started
postgres_1  | CREATE DATABASE
Copy code
postgres_1  | 
postgres_1  | 
postgres_1  | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1  | 
postgres_1  | waiting for server to shut down...2021-01-27 22:21:57.793 UTC [48] LOG:  received fast shutdown request
ui_1        | 👾👾👾 UI running at localhost:8080 👾👾👾
postgres_1  | .2021-01-27 22:21:57.794 UTC [48] LOG:  aborting any active transactions
postgres_1  | 2021-01-27 22:21:57.797 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
postgres_1  | 2021-01-27 22:21:57.798 UTC [50] LOG:  shutting down
postgres_1  | 2021-01-27 22:21:57.816 UTC [48] LOG:  database system is shut down
postgres_1  |  done
postgres_1  | server stopped
postgres_1  | 
postgres_1  | PostgreSQL init process complete; ready for start up.
postgres_1  | 
postgres_1  | 2021-01-27 22:21:57.909 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2021-01-27 22:21:57.909 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2021-01-27 22:21:57.912 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2021-01-27 22:21:57.926 UTC [76] LOG:  database system was shut down at 2021-01-27 22:21:57 UTC
postgres_1  | 2021-01-27 22:21:57.934 UTC [1] LOG:  database system is ready to accept connections
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
towel_1     | {"severity": "INFO", "name": "prefect-server.Lazarus", "message": "Lazarus will start after an initial delay of 19 seconds..."}
towel_1     | {"severity": "INFO", "name": "prefect-server.Scheduler", "message": "Scheduler will start after an initial delay of 113 seconds..."}
towel_1     | {"severity": "INFO", "name": "prefect-server.ZombieKiller", "message": "ZombieKiller will start after an initial delay of 71 seconds..."}
graphql_1   | 
graphql_1   | Running Alembic migrations...
graphql_1   | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1   | INFO  [alembic.runtime.migration] Will assume transactional DDL.
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade  -> 27811b58307b, Create extensions and initial settings
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 27811b58307b -> 72e2cd3e0469, Initial database tables migration
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 72e2cd3e0469 -> c4d792bdd05e, Add flow run idempotency key
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade c4d792bdd05e -> 3398e4807bfb, Add traversal functions
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 3398e4807bfb -> b9086bd4b962, Create message table
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade b9086bd4b962 -> c1f317aa658c, Remove state_id foreign keys
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade c1f317aa658c -> 6611fd0ccc73, Simplify run state update triggers
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 6611fd0ccc73 -> 70528cee0d2b, Add agent persistence
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 70528cee0d2b -> 9cb7539b7363, Add index on agent_id
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 9cb7539b7363 -> e148cf9f1e5b, Add task run name
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade e148cf9f1e5b -> 850b76d44332, Add flow run config
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 850b76d44332 -> 24f10aeee83e, Add label column to flow runs
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 24f10aeee83e -> 3c87ad7e0b71, Add artifact api
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 3c87ad7e0b71 -> 57ac2cb01ac1, Add index for task run names
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 57ac2cb01ac1 -> 7ca57ea2fdff, Add run_config to flow runs and flow groups
graphql_1   | INFO  [alembic.runtime.migration] Running upgrade 7ca57ea2fdff -> 9116e81c6dc2, Add description to flow group table
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
graphql_1   | Applied Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1   | 
graphql_1   | Database upgraded!
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
graphql_1   | INFO:     Started server process [1]
graphql_1   | INFO:     Waiting for application startup.
graphql_1   | INFO:     Application startup complete.
graphql_1   | INFO:     Uvicorn running on <http://0.0.0.0:4201> (Press CTRL+C to quit)
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
graphql_1   | INFO:     172.22.0.5:34002 - "GET /health HTTP/1.1" 200 OK
apollo_1    | {"status":"ok","version":"master"}
apollo_1    | GraphQL service healthy!
Copy code
apollo_1    | 
apollo_1    | > @ serve /apollo
apollo_1    | > node dist/index.js
apollo_1    | 
apollo_1    | Building schema...
graphql_1   | INFO:     172.22.0.5:34020 - "POST /graphql/ HTTP/1.1" 200 OK
apollo_1    | Building schema complete!
apollo_1    | Server ready at <http://0.0.0.0:4200> 🚀 (version: master)
apollo_1    | Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"71fbeafd-6199-4f80-866c-c78af0c8e676","prefect_server_version":"master","api_version":"0.2.0"}}
graphql_1   | INFO:     172.22.0.5:34036 - "POST /graphql/ HTTP/1.1" 200 OK
graphql_1   | INFO:     172.22.0.5:34048 - "POST /graphql/ HTTP/1.1" 200 OK

                                            WELCOME TO
  
   _____  _____  ______ ______ ______ _____ _______    _____ ______ _______      ________ _____
  |  __ \|  __ \|  ____|  ____|  ____/ ____|__   __|  / ____|  ____|  __ \ \    / /  ____|  __ \
  | |__) | |__) | |__  | |__  | |__ | |       | |    | (___ | |__  | |__) \ \  / /| |__  | |__) |
  |  ___/|  _  /|  __| |  __| |  __|| |       | |     \___ \|  __| |  _  / \ \/ / |  __| |  _  /
  | |    | | \ \| |____| |    | |___| |____   | |     ____) | |____| | \ \  \  /  | |____| | \ \
  |_|    |_|  \_\______|_|    |______\_____|  |_|    |_____/|______|_|  \_\  \/   |______|_|  \_\

    
   Visit <http://localhost:8080> to get started, or check out the docs at <https://docs.prefect.io>
The UI was working fine 3 days ago
I am fairly new to deveops and data engineering, so my apologies if I'm missing something obvious here
z
Hi @Peyton Runyan — it looks like the database got created anew, can you make sure a tenant has been created?
prefect backend server && prefect server create-tenant --name default --slug default
(it should be more than blank if the tenant is missing but I’m not sure what else would cause this)
p
Copy code
(venv) (base) ➜  prefect git:(master) ✗ prefect backend server && prefect server create-tenant --name default --slug default
Backend switched to server
Traceback (most recent call last):
  File "/Users/peytonrunyan/TRP/prefect/venv/bin/prefect", line 33, in <module>
    sys.exit(load_entry_point('prefect', 'console_scripts', 'prefect')())
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peytonrunyan/TRP/prefect/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/peytonrunyan/TRP/prefect-open-source/prefect/src/prefect/cli/server.py", line 467, in create_tenant
    tenant_id = client.create_tenant(name=name, slug=slug)
  File "/Users/peytonrunyan/TRP/prefect-open-source/prefect/src/prefect/client/client.py", line 140, in create_tenant
    variables=dict(input=dict(name=name, slug=slug)),
  File "/Users/peytonrunyan/TRP/prefect-open-source/prefect/src/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': 'Uniqueness violation.', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Uniqueness violation.'}}}]
z
Well that means one exists 🙂 great. Can you open the developer console in your browser and see if there are error messages?
p
Thanks for your time on this!
z
Ah this looks like an issue with our new SSO work… I’ll ping someone who knows more.
🙏 1
Can you try switching to a release tag instead of the master branch? The Okta code shouldn’t be released yet. 
git checkout 0.14.5
p
oh whoops - that master is my master, not prefect master. Here's the version installed in my venv: 
prefect 0.14.2+35.ge2edc8aeb
Just did a pip upgrade and will try now 
Successfully installed prefect-0.14.5
✅ 1
that solved it - thanks!
z
Wonderful! Since the prefect version was 
+35
commits ahead of a release tag it was pulling the 
master
branch of the UI which is normally fine but since it’s got some breaking changes right now it won’t work. You can override the UI tag (see https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/server.py#L323-L331) or just make sure you’re on a tagged commit of 
prefect
in the future 🙂
🙌 1
🙏 1
✅ 1
s
@Zanie I am also getting the same issue I have tried with both prefecthq/ui:core-0.14.4 an d prefecthq/ui:core-0.14.5 images in AWS ECS Fargate setup and I am also getting empty screen on. prefect-server ui and on running tenant creation code I am getting the below errors like @Peyton Runyan
Copy code
(.venv) sagungargs@Saguns-MacBook-Pro flows % prefect server create-tenant --name default --slug default
Traceback (most recent call last):
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/sagungargs/.pyenv/versions/3.7.7/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/sagungargs/.pyenv/versions/3.7.7/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/sagungargs/.pyenv/versions/3.7.7/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/sagungargs/.pyenv/versions/3.7.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Users/sagungargs/.pyenv/versions/3.7.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1116434d0>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1116434d0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sagungargs/Work/portcast2/flows/.venv/bin/prefect", line 10, in <module>
    sys.exit(cli())
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/cli/server.py", line 467, in create_tenant
    tenant_id = client.create_tenant(name=name, slug=slug)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/client/client.py", line 140, in create_tenant
    variables=dict(input=dict(name=name, slug=slug)),
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/client/client.py", line 304, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/client/client.py", line 220, in post
    retry_on_api_error=retry_on_api_error,
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/client/client.py", line 446, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/prefect/client/client.py", line 346, in _send_request
    response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=15)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1116434d0>: Failed to establish a new connection: [Errno 61] Connection refused'))
z
Hi this error looks different -- it looks like you can't connect to your server. Can you navigate to 
<http://localhost:4200/graphql>
in your browser?
s
@Zanie thanks, I was able to go past this error by setting
Copy code
$export PREFECT__USER_CONFIG_PATH= "$Home/.prefect/config.toml" where the $HOME I generated using the $pwd command
But @Zanie Now I am running into issues with creating a tenant. It has been 3 days since I am unable to resolve it. Based on prefect-graphql container logs, I think I am realizing it could be haura-admin-secret issue as mutations are failing while creating a tenant. I am not sure which containers require ENV variable set to make it work
Copy code
prefect.utilities.exceptions.ClientError: [{'message': '[Errno -2] Name or service not known', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_tenant'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '[Errno -2] Name or service not known'}}}]
@Zanie Do you think this is because I am using hasura/graphql-engine:v1.3.3 instead of hasura/graphql-engine:v1.3.0 [ I see there is a breaking change mentioned by hasura repo for treatment of headers in graphql after v1.3.0] Could this actually explain why I am getting above errors vhttps://github.com/hasura/graphql-engine/releases.
z
If you've modified the docker-compose file to be using unsupported versions of services I'm not sure I can help.
Please open a new thread so you're not spamming the original user who created this one and provide the full details fo your setup.
s
Apologies. Noted thanks @Zanie
38 Views
Open in Slack
PreviousNext
Powered by