https://prefect.io logo
#prefect-server
Title
# prefect-server
c

Chris Bowen

05/28/2021, 4:45 PM
Hello, I'm very new to Prefect. I have it running on a RHL server, but I'm running into a couple behaviors I'm not able to find answers for via googling- hoping to get some advice. First thing: when I initially did the tutorial/installation and ran
prefect server start
, the task submitted and didn't run interactively. I'm not sure if I inadvertently changed a setting, but for the past day when I start the server, it locks up the terminal and runs as an interactive job. I don't see a setting anywhere to modify it and run it as a daemon or comparable solution. Any ideas? I realize this might b e a Linux issue more than a Prefect one. Second thing: I can't get an agent to start. I'm getting a timeout every time I try to fire it up. I can access the UI from a browser on port 8081 outside of the RHL VM that the server is running on. I can also access the utility that runs on port 4200 from a browser. For reference, there is also an Airflow instance running on this RHL server, so I've been using this command to start prefect:
prefect server start --postgres-port=5433 --ui-port=8081
. I haven't found documentation, troubleshooting guides, or anybody asking online about the agent timing out. I tried running the start agent job as the root user and it still times out, so I don't think it's permission, but I'm happy to check anything. I did run
prefect backend server
. Here's the agent command:
Copy code
prefect agent local start
I did try messing around with the various hostname flags for this command, I believe I tried every port displayed in the docker list below. I think the agent should be coming up on 4200? Here's the docker info for the Prefect containers:
Copy code
d343f6b8e7a5   prefecthq/ui:core-0.14.20       "/docker-entrypoint.…"   8 minutes ago   Up 8 minutes (healthy)   80/tcp, 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp   tmp_ui_1
9c993c53daed   prefecthq/apollo:core-0.14.20   "tini -g -- bash -c …"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:4200->4200/tcp, :::4200->4200/tcp           tmp_apollo_1
f17c041a6ac8   prefecthq/server:core-0.14.20   "tini -g -- python s…"   8 minutes ago   Up 8 minutes                                                                 tmp_towel_1
65cd55a58004   prefecthq/server:core-0.14.20   "tini -g -- bash -c …"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:4201->4201/tcp, :::4201->4201/tcp           tmp_graphql_1
9a251fd9322f   hasura/graphql-engine:v1.3.3    "graphql-engine serve"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp           tmp_hasura_1
fd22ad83e1a5   postgres:11                     "docker-entrypoint.s…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:5433->5432/tcp, :::5433->5432/tcp           tmp_postgres_1
Here's the command I'm executing for the agent:
Copy code
$ prefect agent local start
/usr/local/lib/python3.8/site-packages/prefect/tasks/__init__.py:8: UserWarning: SQLite tasks require sqlite3 to be installed
  import prefect.tasks.database
[2021-05-28 16:32:21,594] INFO - agent | Registering agent...
And here's the error:
Copy code
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)')
...
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)'))
Here's the installation guide I followed: https://docs.prefect.io/core/getting_started/installation.html Any help or advice is much appreciated, thank you!
k

Kevin Kho

05/28/2021, 4:58 PM
Hey @Chris Bowen, for the first item, you want
prefect server start --detach
. For the second one, you might need to set up the API IP in your
config.toml
file like
Copy code
[server]
host = "<http://localhost>"
Maybe this will help
And this
Btw, some people are not aware so just making sure you are, it’s entirely free to get started with Prefect Cloud and everyone gets 10,000 free task runs to start.
c

Chris Bowen

05/28/2021, 5:09 PM
I saw that on the pricing, appreciate you letting me know. Right now I do need to try to get the local version working given some other requirements. Thanks so much for the response. For the backend.toml file, should it look something like this?
Copy code
backend = "server"

[server]
host = "<http://localhost>"

[api]
host = "<http://localhost>"
port = "5433"
url = "<http://localhost:5433>"
I'm not 100% clear which container is the "API" container that the agent is trying to access.
Interesting too, even with the --detach, it seems to not release the terminal:
Copy code
$ prefect server start --postgres-port=5433 --ui-port=8081 --detach
/usr/local/lib/python3.8/site-packages/prefect/tasks/__init__.py:8: UserWarning: SQLite tasks require sqlite3 to be installed
  import prefect.tasks.database
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling apollo   ... done
Pulling towel    ... done
Pulling ui       ... done
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1   ... done
Creating tmp_graphql_1  ... done
Creating tmp_apollo_1   ... done
Creating tmp_towel_1    ... done
Creating tmp_ui_1       ... done
Just hangs (everything is running), but if I to a
ctrl + c
it kills the containers. Unless I'm not giving it long enough to detach, but I've waited 5-10 minutes. Thanks again for your help.
k

Kevin Kho

05/28/2021, 5:19 PM
Oh sorry that may have been confusing. You only need the
server
for portion then just host and port I think.
c

Chris Bowen

05/28/2021, 5:29 PM
OK, makes sense... I've modified the backend.toml file to look like this:
Copy code
backend = "server"

[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"

        [server.database]
        host = "localhost"
        port = "5432"
        host_port ="5433"

        [server.ui]
        host = "<http://localhost>"
        port = "8080"
        host_port = "8081"
Which mean I only have to run
prefect server start --detach
and it comes up with the UI on 8081 and PostGres on 5433, so that's a nice improvement (and I can tell its picking up the toml file). Strangely, the start job is still locking the terminal. I'm also still getting timed out on
prefect agent local start
Found this on Github which is handy: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml
Just a sanity check, are backend.toml and config.toml different things? I've only got .prefect/backend.toml and I don't see a config.toml
z

Zanie

05/28/2021, 5:43 PM
backend.toml
is intended for automatic writes from the prefect library
You'll want to create a
config.toml
and work with that
c

Chris Bowen

05/28/2021, 7:03 PM
Well, I was able to get the job to detach after restarting the server. Still unable to get the agent to start. I have a
config.toml
that looks like this:
Copy code
backend = "server"

[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"

        [server.database]
        host = "localhost"
        port = "5432"
        host_port ="5433"

        [server.ui]
        host = "<http://localhost>"
        port = "8080"
        host_port = "8081"
I set my
backend.toml
to its original state, so it just contains the line
backend = "server"
Not sure what else to mess with to get the agent cooperating.
z

Zanie

05/28/2021, 7:03 PM
Are you running the agent on the same machine as the server?
You'll want to change
Copy code
[server]
host = "<http://localhost>"
on the agent's machine to point to the server if not
c

Chris Bowen

05/28/2021, 7:08 PM
I am running the agent on the same machine
g

Garret Cook

05/28/2021, 7:21 PM
I had the same error when I started on Centos7, I had to add the docker0 interface to the trusted zone in firewalld. Not sure if that will help in your circumstance.
👍 1
c

Chris Bowen

06/01/2021, 4:50 PM
Figured I would report back with my update- I finally got the Prefect agent to run. I tried adding
docker0
to the trusted zone in
firewalld
, but that resulted in other errors from Docker- it has its own xml file in
/etc/firewalld/zones/
named
docker.xml
Adding
docker0
to the
trusted.xml
settings wound up giving me this error: https://github.com/moby/moby/issues/41609 So, I undid that change and reloaded the default settings for Docker and
firewalld
. But, I did notice that by default the
firewalld
service was not enabled on my server. Upon starting
firewalld
, I was able to boot up the agent successfully without changing the trust zone for
docker0
. This produced another problem, though, as
firewalld
took over management of my network traffic and I was no longer able to access any of the Prefect services from outside of the RHEL VM via a browser. Not sure if this is 100% correct, but I wound up adding all ports for all services to the
public
zone for
firewalld
via:
Copy code
sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
Rinse and repeat for all host ports that the Docker containers are running on. Now I am able to access all of the services on the appropriate ports and the local agent properly registered. Thanks for all the help! Really appreciate it.
z

Zanie

06/01/2021, 4:53 PM
Wow that's a weird one.
Thanks for following up!
c

Chris Bowen

06/01/2021, 4:57 PM
No prob, it is pretty strange. From what I've read, it seems that
iptables
is 'legacy' now and
firewalld
has become the modern approach. I'm guessing that something in
iptables
was not configured to allow the traffic approach the Agent was trying to take to get through, while
firewalld
by default allowed it. I'm not an expert in networking though, so my general approach is get it working and don't think about it too much...
2 Views