Chris Bowen
05/28/2021, 4:45 PMprefect server start
, the task submitted and didn't run interactively. I'm not sure if I inadvertently changed a setting, but for the past day when I start the server, it locks up the terminal and runs as an interactive job. I don't see a setting anywhere to modify it and run it as a daemon or comparable solution. Any ideas? I realize this might b e a Linux issue more than a Prefect one.
Second thing: I can't get an agent to start. I'm getting a timeout every time I try to fire it up. I can access the UI from a browser on port 8081 outside of the RHL VM that the server is running on. I can also access the utility that runs on port 4200 from a browser. For reference, there is also an Airflow instance running on this RHL server, so I've been using this command to start prefect: prefect server start --postgres-port=5433 --ui-port=8081
. I haven't found documentation, troubleshooting guides, or anybody asking online about the agent timing out. I tried running the start agent job as the root user and it still times out, so I don't think it's permission, but I'm happy to check anything. I did run prefect backend server
.
Here's the agent command:
prefect agent local start
I did try messing around with the various hostname flags for this command, I believe I tried every port displayed in the docker list below. I think the agent should be coming up on 4200?
Here's the docker info for the Prefect containers:
d343f6b8e7a5 prefecthq/ui:core-0.14.20 "/docker-entrypoint.…" 8 minutes ago Up 8 minutes (healthy) 80/tcp, 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp tmp_ui_1
9c993c53daed prefecthq/apollo:core-0.14.20 "tini -g -- bash -c …" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:4200->4200/tcp, :::4200->4200/tcp tmp_apollo_1
f17c041a6ac8 prefecthq/server:core-0.14.20 "tini -g -- python s…" 8 minutes ago Up 8 minutes tmp_towel_1
65cd55a58004 prefecthq/server:core-0.14.20 "tini -g -- bash -c …" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:4201->4201/tcp, :::4201->4201/tcp tmp_graphql_1
9a251fd9322f hasura/graphql-engine:v1.3.3 "graphql-engine serve" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp tmp_hasura_1
fd22ad83e1a5 postgres:11 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp tmp_postgres_1
Here's the command I'm executing for the agent:
$ prefect agent local start
/usr/local/lib/python3.8/site-packages/prefect/tasks/__init__.py:8: UserWarning: SQLite tasks require sqlite3 to be installed
import prefect.tasks.database
[2021-05-28 16:32:21,594] INFO - agent | Registering agent...
And here's the error:
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)')
...
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)'))
Here's the installation guide I followed: https://docs.prefect.io/core/getting_started/installation.html
Any help or advice is much appreciated, thank you!Kevin Kho
05/28/2021, 4:58 PMprefect server start --detach
. For the second one, you might need to set up the API IP in your config.toml
file like
[server]
host = "<http://localhost>"
Chris Bowen
05/28/2021, 5:09 PMbackend = "server"
[server]
host = "<http://localhost>"
[api]
host = "<http://localhost>"
port = "5433"
url = "<http://localhost:5433>"
I'm not 100% clear which container is the "API" container that the agent is trying to access.$ prefect server start --postgres-port=5433 --ui-port=8081 --detach
/usr/local/lib/python3.8/site-packages/prefect/tasks/__init__.py:8: UserWarning: SQLite tasks require sqlite3 to be installed
import prefect.tasks.database
Pulling postgres ... done
Pulling hasura ... done
Pulling graphql ... done
Pulling apollo ... done
Pulling towel ... done
Pulling ui ... done
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1 ... done
Creating tmp_graphql_1 ... done
Creating tmp_apollo_1 ... done
Creating tmp_towel_1 ... done
Creating tmp_ui_1 ... done
Just hangs (everything is running), but if I to a ctrl + c
it kills the containers.
Unless I'm not giving it long enough to detach, but I've waited 5-10 minutes.
Thanks again for your help.Kevin Kho
05/28/2021, 5:19 PMserver
for portion then just host and port I think.Chris Bowen
05/28/2021, 5:29 PMbackend = "server"
[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"
[server.database]
host = "localhost"
port = "5432"
host_port ="5433"
[server.ui]
host = "<http://localhost>"
port = "8080"
host_port = "8081"
Which mean I only have to run prefect server start --detach
and it comes up with the UI on 8081 and PostGres on 5433, so that's a nice improvement (and I can tell its picking up the toml file).
Strangely, the start job is still locking the terminal.
I'm also still getting timed out on prefect agent local start
Found this on Github which is handy: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.tomlZanie
05/28/2021, 5:43 PMbackend.toml
is intended for automatic writes from the prefect libraryconfig.toml
and work with thatChris Bowen
05/28/2021, 7:03 PMconfig.toml
that looks like this:
backend = "server"
[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"
[server.database]
host = "localhost"
port = "5432"
host_port ="5433"
[server.ui]
host = "<http://localhost>"
port = "8080"
host_port = "8081"
I set my backend.toml
to its original state, so it just contains the line backend = "server"
Not sure what else to mess with to get the agent cooperating.Zanie
05/28/2021, 7:03 PM[server]
host = "<http://localhost>"
on the agent's machine to point to the server if notChris Bowen
05/28/2021, 7:08 PMGarret Cook
05/28/2021, 7:21 PMChris Bowen
06/01/2021, 4:50 PMdocker0
to the trusted zone in firewalld
, but that resulted in other errors from Docker- it has its own xml file in /etc/firewalld/zones/
named docker.xml
Adding docker0
to the trusted.xml
settings wound up giving me this error: https://github.com/moby/moby/issues/41609
So, I undid that change and reloaded the default settings for Docker and firewalld
.
But, I did notice that by default the firewalld
service was not enabled on my server. Upon starting firewalld
, I was able to boot up the agent successfully without changing the trust zone for docker0
.
This produced another problem, though, as firewalld
took over management of my network traffic and I was no longer able to access any of the Prefect services from outside of the RHEL VM via a browser.
Not sure if this is 100% correct, but I wound up adding all ports for all services to the public
zone for firewalld
via:
sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
Rinse and repeat for all host ports that the Docker containers are running on.
Now I am able to access all of the services on the appropriate ports and the local agent properly registered.
Thanks for all the help! Really appreciate it.Zanie
06/01/2021, 4:53 PMChris Bowen
06/01/2021, 4:57 PMiptables
is 'legacy' now and firewalld
has become the modern approach. I'm guessing that something in iptables
was not configured to allow the traffic approach the Agent was trying to take to get through, while firewalld
by default allowed it. I'm not an expert in networking though, so my general approach is get it working and don't think about it too much...