https://prefect.io logo
Powered by
Title
q

Questionnaire

05/21/2020, 10:41 AM
Hello folks, I'm trying to register my flow but my API status in local UI remains Attempting to connect... I'm sharing 
config.toml
and UI screenshot. Please help. and getting an error.
config.toml
=> 
[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"
endpoint = "${server.host}:${server.port}"

	[server.database]
	host = "localhost"
	port = "5432"
	host_port = "5432"
	name = "dev"
	username = "root"
	password = "password"
	connection_url = "postgresql://${server.database.username}:${server.database.password}@${server.database.port}/${server.database.name}"

	[server.graphql]
	host = "0.0.0.0"
	port = "4201"
	host_port = "4201"
	debug = true
	path = "/graphql/"

	[server.hasura]
	host = "localhost"
    port = "3000"
    host_port = "3000"
	admin_secret = ""
	claims_namespace = "hasura-claims"
	graphql_url = "http://${server.hasura.host}:${server.hasura.port}/v1alpha1/graphql"
	ws_url = "ws://${server.hasura.host}:${server.hasura.port}/v1alpha1/graphql"
	execute_retry_seconds = 10

	[server.ui]
	host = "<http://localhost>"
	port = "8080"
	host_port = "8080"
	endpoint = "${server.ui.host}:${server.ui.port}"
error
=> 
Result check: OK
Traceback (most recent call last):
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tanmay/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/tanmay/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    flow.register()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/core/flow.py", line 1429, in register
    no_url=no_url,
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/client.py", line 640, in register
    version_group_id=version_group_id,
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/client.py", line 218, in graphql
    token=token,
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/client.py", line 178, in post
    token=token,
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/client.py", line 304, in _request
    response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
  File "/home/tanmay/.local/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
@Jim Crist-Harif can you help me in this?
l

Laura Lorenz (she/her)

05/21/2020, 12:30 PM
Hello @Questionnaire! My guess would be that the graphql service is not up, how are you starting the rest of the services for server? Docker? If so what is the output of 
docker ps
, or one step further, of the 
docker logs
command against the graphql container?
q

Questionnaire

05/21/2020, 12:33 PM
hey @Laura Lorenz (she/her) , I'm doing 
prefect server start
yeah which lifts docker file only, So, 
docker ps
outputs: 
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                     NAMES
eaa352a76c5a        prefecthq/ui:0.11.2            "/intercept.sh"          2 hours ago         Up 2 hours          80/tcp, 0.0.0.0:8080->8080/tcp            tmp_ui_1
63ea0676e7da        prefecthq/server:0.11.2        "python src/prefect_…"   2 hours ago         Up 2 hours                                                    tmp_scheduler_1
1fd04acd8f1c        prefecthq/apollo:0.11.2        "npm run serve"          2 hours ago         Up 2 hours          0.0.0.0:4200->4200/tcp                    tmp_apollo_1
4fb46e7d56ec        prefecthq/server:0.11.2        "bash -c 'prefect-se…"   2 hours ago         Up About a minute   0.0.0.0:4201->4201/tcp                    tmp_graphql_1
0d82aa8247ee        hasura/graphql-engine:v1.1.0   "graphql-engine serve"   2 hours ago         Up About a minute   0.0.0.0:3000->3000/tcp                    tmp_hasura_1
c432bfe4de63        postgres:11                    "docker-entrypoint.s…"   2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp
@Laura Lorenz (she/her) 
docker logs
outputs: 
{"type":"pg-client","timestamp":"2020-05-21T12:39:27.199+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
{"type":"pg-client","timestamp":"2020-05-21T12:41:38.206+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
{"internal":"could not connect to server: Connection timed out\n\tIs the server running on host \"5432\" (0.0.21.56) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
{"type":"startup","timestamp":"2020-05-21T12:41:44.529+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}}
{"type":"startup","timestamp":"2020-05-21T12:41:44.529+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"<postgresql://root>:...@5432/data_model_dev"}}}
l

Laura Lorenz (she/her)

05/21/2020, 1:36 PM
Cool thanks @Questionnaire! From the graphql logs it looks like graphql can’t start because it can’t connect to postgres. Either the postgres container is sick, or possibly the configuration is off. Is there any weird output from your postgres container’s logs that indicate it wasn’t able to start up all the way? Can you connect to your postgres container with the credentials from your config.toml (with another client like with pgadmin GUI or psql?)
q

Questionnaire

05/21/2020, 1:37 PM
Let me check one
@Laura Lorenz (she/her) I confirmed the connection is settuped using client and ran a postgres task also using 
flow.run()
method.
Hey @Laura Lorenz (she/her), Let me tell you what I did: 
pip install prefect
Put 
config.toml
 in 
~/.prefect
Then 
prefect backend server
 after that 
prefect server start
. And my flow is: 
import prefect
from prefect import task, Flow

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

flow = Flow("hello-flow", tasks=[hello_task])
flow.register()
and my 
.toml
is already here in thread for ur reference.
l

Laura Lorenz (she/her)

05/21/2020, 2:59 PM
@Questionnaire Awesome thank you! With all that I can try and reproduce with your situation locally for me as a next step. I’,m not 100% sure this is the issue but I feel like something is up between how 
prefect server start
configured your postgres server against your configuration: I’m suspicious because config.toml specifies that the databsae should be called 
dev
but the graphql output looks like its trying to access a db called 
data_model_dev
Ok I can reproduce your issue! Going to mess around with it for a bit
@Questionnaire Ok I think it’s a typo in your config.toml! server.database.connection_url is missing the host interpolation right now it is: 
connection_url = "postgresql://${server.database.username}:${server.database.password}@${server.database.port}/${server.database.name}"
but it should be: 
connection_url = "postgresql://${server.database.username}:${server.database.password}@${server.database.host}:${server.database.port}/${server.database.name}"
If I change that then I go from having your same issue, to it all working! I do think you might have an env var that is overwriting your database name in your config.toml in your local terminal, based on your logs. If you run 
env | grep -i data_model_dev
in the terminal you were starting prefect server from, are there any matches?
q

Questionnaire

05/21/2020, 3:23 PM
env | grep -i data_model_dev
it returns me nothing on terminal
l

Laura Lorenz (she/her)

05/21/2020, 3:24 PM
Ok maybe I’m paranoid then 😅, if you don’t mind updating your config.toml with the new connection_url and giving it a try, we can just see if it’s ok now and if not, send me some the new docker logs please 🙂
q

Questionnaire

05/21/2020, 3:30 PM
AHH, no no I updated it also .... @Laura Lorenz (she/her)
Is it necessary to name dbname= 
dev
???
l

Laura Lorenz (she/her)

05/21/2020, 3:34 PM
Technically the only necessary thing is to have config.toml to have the name the same as a database that exists in your postgres container. FWIW when the postgres container first starts up from an image it will create a database off of the name that is in config.toml at that time so if you are changing config.toml between container starts it might be confusing it.
q

Questionnaire

05/21/2020, 3:39 PM
Yeah, so I've db in my postgres with name 
data_model_dev
that's why I put the name as same.... I was just asking to confirm that is this not the case where graphql only search for 
dev
l

Laura Lorenz (she/her)

05/21/2020, 3:41 PM
Gotcha, no it’s not hardcoded, it will pull it off of your config.toml’s server.database.name, so even if you made a database called 
super_awesome_database
independently in your postgres instance, if you change config.toml and restart graphql it will connect to 
super_awesome_database
instead
q

Questionnaire

05/21/2020, 3:43 PM
yeah @Laura Lorenz (she/her), So now my docker logs is saying this: 
":{"x-hasura-role":"admin"},"request_id":"fb1e6596-af31-4e13-9987-b715d7cac239","response_size":40,"request_read_time":1.2532e-5},"http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1alpha1/graphql","ip":"192.168.48.5","method":"POST","content_encoding":"gzip"}}}
{"type":"http-log","timestamp":"2020-05-21T15:42:49.798+0000","level":"info","detail":{"operation":{"query_execution_time":5.558238e-3,"user_vars":{"x-hasura-role":"admin"},"request_id":"cc3eb829-b154-4570-961c-b9bbfe332ad3","response_size":44,"request_read_time":9.29e-6},"http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1alpha1/graphql","ip":"192.168.48.5","method":"POST","content_encoding":"gzip"}}}
and api is connected also
yaay
l

Laura Lorenz (she/her)

05/21/2020, 4:02 PM
Yay awesome!!!
q

Questionnaire

05/21/2020, 4:11 PM
@Laura Lorenz (she/her) you're STAR🌟
@Laura Lorenz (she/her) Can you just tell me one thing like I've to pass parameters in my flow... How will I pass them after registering?
l

Laura Lorenz (she/her)

05/21/2020, 4:29 PM
@Questionnaire Depends how you are going to start the flow runs! You can pass them when you call flow.run() if you are starting it from a python script (see https://docs.prefect.io/core/concepts/parameters.html#parameters), when you start the flow from the UI (the “Run” button NOT the “Quick Run” button - see https://docs.prefect.io/orchestration/ui/flow.html#run), or you can add default parameters to your clocks if you are going to schedule the flow (see https://docs.prefect.io/core/concepts/schedules.html#complex-schedules).
q

Questionnaire

05/21/2020, 6:17 PM
Ok, @Laura Lorenz (she/her) 
flow.register()
CronClock('0 0 * * *', parameter_defaults={
  "key:: "value"
}).events()
Is this a right way to add it???
l

Laura Lorenz (she/her)

05/21/2020, 6:29 PM
You need to attach the schedule to the flow before registering it. It looks like there is some syntax errors in your parameter defaults dictionary, and you don’t need to call the events method too. Here’s an example with an IntervalClock with no parameters, but hopefully it helps with the gist: https://github.com/PrefectHQ/prefect/blob/master/examples/github_release_cycle.py#L40-L45
q

Questionnaire

05/22/2020, 2:18 PM
Thanks @Laura Lorenz (she/her) 🙂
#prefect-communityJoin Slack
Powered by