https://prefect.io logo
Join Slack
Powered by
After starting prefect-server, I get this error in...
# prefect-ui
p
After starting prefect-server, I get this error in the logs.
Copy code
apollo_1     | 2021-07-02T15:47:56.760Z {"message":"Cannot query field \"api\" on type \"Query\".","locations":[{"line":2,"column":3}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
When I access the gui, website partially loads
I was trying to debug this the other day, but ran into my earlier problem (port 3000 allocation error)
k
Do you get that error immediately? Even before you visit the UI?
p
Copy code
postgres_1   | PostgreSQL init process complete; ready for start up.
postgres_1   | 
postgres_1   | 2021-07-02 15:53:27.338 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1   | 2021-07-02 15:53:27.338 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1   | 2021-07-02 15:53:27.340 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1   | 2021-07-02 15:53:27.351 UTC [75] LOG:  database system was shut down at 2021-07-02 15:53:27 UTC
postgres_1   | 2021-07-02 15:53:27.355 UTC [1] LOG:  database system is ready to accept connections
apollo_1     | 2021-07-02T15:53:46.703Z {"message":"Cannot query field \"api\" on type \"Query\".","locations":[{"line":2,"column":3}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
apollo_1     | 2021-07-02T15:53:47.810Z {"message":"Cannot query field \"api\" on type \"Query\".","locations":[{"line":2,"column":3}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
apollo_1     | 2021-07-02T15:53:53.214Z {"message":"Cannot query field \"api\" on type \"Query\".","locations":[{"line":2,"column":3}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
^CGracefully stopping... (press Ctrl+C again to force)
Exception caught; killing services (press ctrl-C to force)
Killing t_ui_1         ... 
Killing t_ui_1         ... done
Killing t_apollo_1     ... 
Killing t_apollo_1     ... done
Killing t_hasura_1     ... done
Killing t_postgres_1   ... done
ERROR: 25: open failed: connect failed: Connection refused
Stopping t_ui_1       ... done
Stopping t_apollo_1   ... done
Stopping t_graphql_1  ... done
Stopping t_hasura_1   ... done
Stopping t_postgres_1 ... done
Removing t_ui_1        ... done
Removing t_scheduler_1 ... done
Removing t_apollo_1    ... done
Removing t_graphql_1   ... done
Removing t_hasura_1    ... done
Removing t_postgres_1  ... done
Removing network prefect-server
k
Ok I’ll check with the team
p
let me try to capture the entire log
Unfortunately, I'm not able to get the GRAPHQL_VALIDATION_FAILED error i reported but when i access localhost:8080, the same issue with graphgl occurs
graphgl_prefect_logs.txt
I'm trying to see if i can register a flow
k
Hey @peter zhao, this looks like the database didn’t spin up right. The feedback is to try again or to start the database separately, and then the other services. You can see this for that
p
Copy code
graphql_1    | Database upgraded!                                                                                                                                               
graphql_1    | INFO:     Started server process [1]                                                                                                                             
graphql_1    | INFO:     Waiting for application startup.                                                                                                                       
graphql_1    | INFO:     Application startup complete.                                                                                                                          
graphql_1    | INFO:     Uvicorn running on <http://0.0.0.0:4201> (Press CTRL+C to quit)                                                                                          
graphql_1    | INFO:     172.21.0.6:60972 - "GET /health HTTP/1.1" 200 OK                                                                                                       
graphql_1    | INFO:     172.21.0.6:60976 - "POST /graphql/ HTTP/1.1" 200 OK                                                                                                    
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: FetchError: request to <https://sens-o-matic.prefect.io/> failed, reason: unable to get local issuer certificate       
apollo_1     |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1455:11)                                                                          
apollo_1     |     at ClientRequest.emit (events.js:182:13)                                                                                                                     
apollo_1     |     at TLSSocket.socketErrorListener (_http_client.js:392:9)                                                                                                     
apollo_1     |     at TLSSocket.emit (events.js:182:13)                                                                                                                         
apollo_1     |     at emitErrorNT (internal/streams/destroy.js:82:8)                                                                                                            
apollo_1     |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)                                                                                                    
apollo_1     |     at process._tickCallback (internal/process/next_tick.js:63:19)                                                                                               
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch blo
ck, or by rejecting a promise which was not handled with .catch(). (rejection id: 8)                                                                                            
apollo_1     | 2021-07-02T17:25:20.202Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"heartbeat","payload":{"id":"6f2877a5-3ada-451d-b30d-
29d52573d458"}}                                                                                                                                                                 
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: FetchError: request to <https://sens-o-matic.prefect.io/> failed, reason: unable to get local issuer certificate       
apollo_1     |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1455:11)                                                                          
apollo_1     |     at ClientRequest.emit (events.js:182:13)                                                                                                                     
apollo_1     |     at TLSSocket.socketErrorListener (_http_client.js:392:9)                                                                                                     
apollo_1     |     at TLSSocket.emit (events.js:182:13)                                                                                                                         
apollo_1     |     at emitErrorNT (internal/streams/destroy.js:82:8)                                                                                                            
apollo_1     |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)                                                                                                    
apollo_1     |     at process._tickCallback (internal/process/next_tick.js:63:19)                                                                                               
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch blo
ck, or by rejecting a promise which was not handled with .catch(). (rejection id: 9)                                                                                            
apollo_1     | 2021-07-02T17:35:19.782Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"heartbeat","payload":{"id":"6f2877a5-3ada-451d-b30d-
29d52573d458"}}                                                                                                                                                                 
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: FetchError: request to <https://sens-o-matic.prefect.io/> failed, reason: unable to get local issuer certificate       
apollo_1     |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1455:11)                                                                          
apollo_1     |     at ClientRequest.emit (events.js:182:13)                                                                                                                     
apollo_1     |     at TLSSocket.socketErrorListener (_http_client.js:392:9)                                                                                                     
apollo_1     |     at TLSSocket.emit (events.js:182:13)                                                                                                                         
apollo_1     |     at emitErrorNT (internal/streams/destroy.js:82:8)                                                                                                            
apollo_1     |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)                                                                                                    
apollo_1     |     at process._tickCallback (internal/process/next_tick.js:63:19)                                                                                               
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch blo
ck, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)                                                                                           
apollo_1     | 2021-07-02T17:45:19.624Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"heartbeat","payload":{"id":"6f2877a5-3ada-451d-b30d-
29d52573d458"}}                                                                                                                                                                 
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: FetchError: request to <https://sens-o-matic.prefect.io/> failed, reason: unable to get local issuer certificate       
apollo_1     |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1455:11)                                                                          
apollo_1     |     at ClientRequest.emit (events.js:182:13)                                                                                                                     
apollo_1     |     at TLSSocket.socketErrorListener (_http_client.js:392:9)                                                                                                     apollo_1     |     at TLSSocket.emit (events.js:182:13)                                                                                                                         
apollo_1     |     at emitErrorNT (internal/streams/destroy.js:82:8)                                                                                                            apollo_1     |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)                                                                                                    apollo_1     |     at process._tickCallback (internal/process/next_tick.js:63:19)                                                                                               
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 11)                                                                                           apollo_1     | 2021-07-02T17:55:19.270Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"heartbeat","payload":{"id":"6f2877a5-3ada-451d-b30d-
29d52573d458"}}                                                                                                                                                                 apollo_1     | (node:25) UnhandledPromiseRejectionWarning: FetchError: request to <https://sens-o-matic.prefect.io/> failed, reason: unable to get local issuer certificate       apollo_1     |     at ClientRequest.<anonymous> (/apollo/node_modules/node-fetch/lib/index.js:1455:11)                                                                          
apollo_1     |     at ClientRequest.emit (events.js:182:13)
apollo_1     |     at TLSSocket.socketErrorListener (_http_client.js:392:9)
apollo_1     |     at TLSSocket.emit (events.js:182:13)
apollo_1     |     at emitErrorNT (internal/streams/destroy.js:82:8)
apollo_1     |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
apollo_1     |     at process._tickCallback (internal/process/next_tick.js:63:19)
apollo_1     | (node:25) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch blo
ck, or by rejecting a promise which was not handled with .catch(). (rejection id: 12)
^ i noticed some exceptions Currently unable to start agent via
Copy code
prefect agent local start
ill try the database solution you proposed
k
Are your errors there also when you do 
prefect backend  cloud
? Just making sure this is 
prefect backend server
?
p
yes
postgre_prefect_logs.txt
postgre_prefect_logs.txt
heres the logs in docker for postgres
i tried creating an external database with the instuctions provided - getting the same error in gui 'api status - attempting to connect... http://localhost:4200/graphql/ I tried deleting all the images and containers on docker (hoping to do a hard reset) and issue is still presists
k
Will ask the team
Do you see any logs about apollo?
Added to the ones above?
p
Here's the apollo logs
apollo_prefect_logs.txt
k
The recommendation is to disable telemetry. Could you try this ?
p
Hey Kevin, I've disabled telemetry and Prefect looks to be up and running and API seems to connect now.
$ prefect agent local start Usage: prefect agent [OPTIONS] COMMAND [ARGS]... Try 'prefect agent -h' for help. Error: No such command 'local'.```
I'm having issues starting up the prefect agent^^ The prefect GUI seems like it's changed as well.
k
What version are you on? That’s really weird
Your UI looks good. This is what happens after you configure the API endpoint.
p
0.12.4
k
Oh I think that CLI interface maybe 0.14 and above? It might be 
prefect local agent start
in that version?
Are you just starting out? 0.12.4 is quite a few versions back. You may wanna upgrade if you can.
p
yes. Let me update everything
both didnt work
Copy code
prefect local agent start
prefect agent local start
k
Both didn’t work after updating?
p
i havent updated yet
I was working on 0.12.4 and everything was working before I encountered this issue
k
I downgraded to test. It’s 
prefect agent start local
in that version
p
what version do you recommend?
it looks like 0.14.22 is the last stable release
k
That’s right for server yeah. There was a fix today for 0.15.0 so you can pull the master with 
prefect server start --version master
.
p
Currently, I have an issue trying to start a agent on 0.14.22
Copy code
$ prefect agent local start
[2021-07-06 19:43:46,273] INFO - agent | Registering agent...
Traceback (most recent call last):
  File "/Users/pzhao/miniconda/envs/prefect_test/bin/prefect", line 10, in <module>
    sys.exit(cli())
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/cli/agent.py", line 150, in start
    start_agent(LocalAgent, import_paths=list(import_paths), **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/cli/agent.py", line 103, in start_agent
    agent.start()
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/agent/agent.py", line 182, in start
    self._setup_api_connection()
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/agent/agent.py", line 899, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/agent/agent.py", line 848, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 1703, in register_agent
    agent_config_id=agent_config_id,
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': 'No tenant found.', 'locations': [{'line': 2, 'column': 5}], 'path': ['register_agent'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'No tenant found.'}}}]
here's what the server logs look like when i try to start up agent
Copy code
ui_1        | 172.18.0.7 - - [06/Jul/2021:19:49:14 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
graphql_1   | INFO:     172.18.0.6:47192 - "POST /graphql/ HTTP/1.1" 200 OK
graphql_1   | INFO:     172.18.0.4:43358 - "GET /health HTTP/1.1" 200 OK
graphql_1   | INFO:     172.18.0.6:47454 - "POST /graphql/ HTTP/1.1" 200 OK
graphql_1   | No tenant found.
graphql_1   | 
graphql_1   | GraphQL request:2:3
graphql_1   | 1 | mutation ($input: register_agent_input!) {
graphql_1   | 2 |   register_agent(input: $input) {
graphql_1   |   |   ^
graphql_1   | 3 |     id
graphql_1   | Traceback (most recent call last):
graphql_1   |   File "/prefect-server/src/prefect_server/api/agents.py", line 36, in register_agent
graphql_1   |     tenant_id = await models.Tenant.where({"id": {"_eq": None}}).first().id
graphql_1   | AttributeError: 'coroutine' object has no attribute 'id'
graphql_1   | 
graphql_1   | During handling of the above exception, another exception occurred:
graphql_1   | 
graphql_1   | Traceback (most recent call last):
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
graphql_1   |     return await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
graphql_1   |     result = await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/agents.py", line 18, in resolve_register_agent
graphql_1   |     name=input.get("name"),
graphql_1   |   File "/prefect-server/src/prefect_server/api/agents.py", line 38, in register_agent
graphql_1   |     raise ValueError("No tenant found.")
graphql_1   | ValueError: No tenant found.
graphql_1   | 
graphql_1   | The above exception was the direct cause of the following exception:
graphql_1   | 
graphql_1   | Traceback (most recent call last):
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 674, in await_completed
graphql_1   |     return await completed
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 659, in await_result
graphql_1   |     return_type, field_nodes, info, path, await result
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 733, in complete_value
graphql_1   |     raise result
graphql_1   |   File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
graphql_1   |     return await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
graphql_1   |     result = await result
graphql_1   |   File "/prefect-server/src/prefect_server/graphql/agents.py", line 18, in resolve_register_agent
graphql_1   |     name=input.get("name"),
graphql_1   |   File "/prefect-server/src/prefect_server/api/agents.py", line 38, in register_agent
graphql_1   |     raise ValueError("No tenant found.")
graphql_1   | graphql.error.graphql_error.GraphQLError: No tenant found.
graphql_1   | 
graphql_1   | GraphQL request:2:3
graphql_1   | 1 | mutation ($input: register_agent_input!) {
graphql_1   | 2 |   register_agent(input: $input) {
graphql_1   |   |   ^
graphql_1   | 3 |     id
graphql_1   | INFO:     172.18.0.6:47594 - "POST /graphql/ HTTP/1.1" 200 OK
towel_1     | {"severity": "INFO", "name": "prefect-server.Scheduler", "message": "Scheduled 0 flow runs."}
towel_1     | {"severity": "DEBUG", "name": "prefect-server.Scheduler", "message": "Heartbeat from Scheduler: next run at 2021-07-06T19:52:08+00:00"}
graphql_1   | INFO:     172.18.0.6:47726 - "POST /graphql/ HTTP/1.1" 200 OK
ui_1        | 172.18.0.7 - - [06/Jul/2021:19:49:44 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
graphql_1   | INFO:     172.18.0.4:43842 - "GET /health HTTP/1.1" 200 OK
k
Did you have a tenant in server? Try 
prefect backend server
(if you haven’t yet) then 
prefect server create-tenant -n default
to make a default tenant
p
ok
I tried and here's the error i get when i try strarting the server
Copy code
(prefect_test) pzhao$ prefect backend server
Backend switched to server
(prefect_test) pzhao$ prefect server create-tenant -n default
Traceback (most recent call last):
  File "/Users/pzhao/miniconda/envs/prefect_test/bin/prefect", line 10, in <module>
    sys.exit(cli())
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/cli/server.py", line 733, in create_tenant
    tenant_id = client.create_tenant(name=name, slug=slug)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 140, in create_tenant
    variables=dict(input=dict(name=name, slug=slug)),
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 304, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 220, in post
    retry_on_api_error=retry_on_api_error,
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 460, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/prefect/client/client.py", line 355, in _send_request
    timeout=prefect.context.config.cloud.request_timeout,
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/requests/sessions.py", line 649, in send
    adapter = self.get_adapter(url=request.url)
  File "/Users/pzhao/miniconda/envs/prefect_test/lib/python3.7/site-packages/requests/sessions.py", line 742, in get_adapter
    raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for ':/graphql'
k
Is your server on? Did you update the server version also?
p
Copy code
prefect version
0.14.22
ok, let me start the server normally then try that command
k
I moved to 0.14.22 and i’m trying this myself now
p
the server and agent appear to be working now
I think I'm in a good place now let me recreate everything I had
k
Ok sounds good! No pressure at all but just wanna make sure you know that Prefect Cloud has 10k free task runs every month which is a lot to get started with
6 Views
Open in Slack
PreviousNext
Powered by