Hello, I'm very new to Prefect. I have it running on a RHL server, but I'm running into a couple behaviors I'm not able to find answers for via googling- hoping to get some advice. First thing: when I initially did the tutorial/installation and ran

prefect server start

, the task submitted and didn't run interactively. I'm not sure if I inadvertently changed a setting, but for the past day when I start the server, it locks up the terminal and runs as an interactive job. I don't see a setting anywhere to modify it and run it as a daemon or comparable solution. Any ideas? I realize this might b e a Linux issue more than a Prefect one. Second thing: I can't get an agent to start. I'm getting a timeout every time I try to fire it up. I can access the UI from a browser on port 8081 outside of the RHL VM that the server is running on. I can also access the utility that runs on port 4200 from a browser. For reference, there is also an Airflow instance running on this RHL server, so I've been using this command to start prefect:

prefect server start --postgres-port=5433 --ui-port=8081

. I haven't found documentation, troubleshooting guides, or anybody asking online about the agent timing out. I tried running the start agent job as the root user and it still times out, so I don't think it's permission, but I'm happy to check anything. I did run

prefect backend server

. Here's the agent command:

prefect agent local start

I did try messing around with the various hostname flags for this command, I believe I tried every port displayed in the docker list below. I think the agent should be coming up on 4200? Here's the docker info for the Prefect containers:

d343f6b8e7a5   prefecthq/ui:core-0.14.20       "/docker-entrypoint.…"   8 minutes ago   Up 8 minutes (healthy)   80/tcp, 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp   tmp_ui_1
9c993c53daed   prefecthq/apollo:core-0.14.20   "tini -g -- bash -c …"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:4200->4200/tcp, :::4200->4200/tcp           tmp_apollo_1
f17c041a6ac8   prefecthq/server:core-0.14.20   "tini -g -- python s…"   8 minutes ago   Up 8 minutes                                                                 tmp_towel_1
65cd55a58004   prefecthq/server:core-0.14.20   "tini -g -- bash -c …"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:4201->4201/tcp, :::4201->4201/tcp           tmp_graphql_1
9a251fd9322f   hasura/graphql-engine:v1.3.3    "graphql-engine serve"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp           tmp_hasura_1
fd22ad83e1a5   postgres:11                     "docker-entrypoint.s…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:5433->5432/tcp, :::5433->5432/tcp           tmp_postgres_1

Here's the command I'm executing for the agent:

$ prefect agent local start
/usr/local/lib/python3.8/site-packages/prefect/tasks/__init__.py:8: UserWarning: SQLite tasks require sqlite3 to be installed
  import prefect.tasks.database
[2021-05-28 16:32:21,594] INFO - agent | Registering agent...

And here's the error:

File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)')
...
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1a862b83d0>, 'Connection to localhost timed out. (connect timeout=15)'))

Here's the installation guide I followed: https://docs.prefect.io/core/getting_started/installation.html Any help or advice is much appreciated, thank you!

Hey @Chris Bowen, for the first item, you want

prefect server start --detach

. For the second one, you might need to set up the API IP in your

config.toml

file like

[server]
host = "<http://localhost>"

I saw that on the pricing, appreciate you letting me know. Right now I do need to try to get the local version working given some other requirements. Thanks so much for the response. For the backend.toml file, should it look something like this?

backend = "server"

[server]
host = "<http://localhost>"

[api]
host = "<http://localhost>"
port = "5433"
url = "<http://localhost:5433>"

I'm not 100% clear which container is the "API" container that the agent is trying to access.

Oh sorry that may have been confusing. You only need the

server

for portion then just host and port I think.

OK, makes sense... I've modified the backend.toml file to look like this:

backend = "server"

[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"

        [server.database]
        host = "localhost"
        port = "5432"
        host_port ="5433"

        [server.ui]
        host = "<http://localhost>"
        port = "8080"
        host_port = "8081"

Which mean I only have to run

prefect server start --detach

and it comes up with the UI on 8081 and PostGres on 5433, so that's a nice improvement (and I can tell its picking up the toml file). Strangely, the start job is still locking the terminal. I'm also still getting timed out on

prefect agent local start

Found this on Github which is handy: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml

backend.toml

is intended for automatic writes from the prefect library

Well, I was able to get the job to detach after restarting the server. Still unable to get the agent to start. I have a

config.toml

that looks like this:

backend = "server"

[server]
host = "<http://localhost>"
port = "4200"
host_port = "4200"

        [server.database]
        host = "localhost"
        port = "5432"
        host_port ="5433"

        [server.ui]
        host = "<http://localhost>"
        port = "8080"
        host_port = "8081"

I set my

backend.toml

to its original state, so it just contains the line

backend = "server"

Not sure what else to mess with to get the agent cooperating.

Are you running the agent on the same machine as the server?

I am running the agent on the same machine

I had the same error when I started on Centos7, I had to add the docker0 interface to the trusted zone in firewalld. Not sure if that will help in your circumstance.

Figured I would report back with my update- I finally got the Prefect agent to run. I tried adding

docker0

to the trusted zone in

firewalld

, but that resulted in other errors from Docker- it has its own xml file in

/etc/firewalld/zones/

named

docker.xml

Adding

docker0

to the

trusted.xml

settings wound up giving me this error: https://github.com/moby/moby/issues/41609 So, I undid that change and reloaded the default settings for Docker and

firewalld

. But, I did notice that by default the

firewalld

service was not enabled on my server. Upon starting

firewalld

, I was able to boot up the agent successfully without changing the trust zone for

docker0

. This produced another problem, though, as

firewalld

took over management of my network traffic and I was no longer able to access any of the Prefect services from outside of the RHEL VM via a browser. Not sure if this is 100% correct, but I wound up adding all ports for all services to the

public

zone for

firewalld

via:

sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp

Rinse and repeat for all host ports that the Docker containers are running on. Now I am able to access all of the services on the appropriate ports and the local agent properly registered. Thanks for all the help! Really appreciate it.

Wow that's a weird one.

No prob, it is pretty strange. From what I've read, it seems that

iptables

is 'legacy' now and

firewalld

has become the modern approach. I'm guessing that something in

iptables

was not configured to allow the traffic approach the Agent was trying to take to get through, while

firewalld

by default allowed it. I'm not an expert in networking though, so my general approach is get it working and don't think about it too much...