https://prefect.io logo
Title
a

Aurélien Vallée

04/14/2021, 6:09 AM
Hello everyone, I am having a somewhat hard time trying to configure a self hosted server for prefect. Most of my struggle is based on what is the proper way of doing things. I started by using
prefect server start
but ran with a bunch of issues: • Most container services are listening on 0.0.0.0, which is not desirable in my case. I have a specific IP on which I would like exposed service to listen. • I am confused as to which services actually need to be exposed beyond the loopback. My understanding is that the UI (of course) needs to be, as well as the GraphQL endpoint (since the UI JS seems to communicate with it). What about the others? I have to admit, I am foreign to hasura/apollo/etc. Since I wanted to customize that, I figured I should somehow generate the
docker-compose.yml
with
prefect server config
and tweak that accordingly, which I did. That did not seem to be a good solution though, since even after tweaking the published ports and trial and error with the various configurations in
docker-compose.yml
, the UI is still trying to reach
<http://localhost:4200>
, even though I tried to tweak the config to have it use
http://<server IP>:4200/
so I guess everything is not configurable through editing the
docker-compose.yml
. I spent quite some bunch of time reading various threads on internet, and it seems to me now the proper way of configuring the server is not to tweak the
docker-compose.yml
but rather to tweak
config.toml
, and
prefect server start
will use that when generating its temporary
docker-compose.yml
. So I did that, but I'm still confused as to what exactly should be exposed and what should not. I also get lost in the options of
prefect server start
, it allows me to no publish container listened ports to the host, but of course them of them should be for the UI/agent/graphql to be available. Not sure I can even articulate the questions I have precisly, I'm just lost in the proper way to configure the server and would need some guidance.
n

nicholas

04/14/2021, 6:28 AM
Hi @Aurélien Vallée - I'd encourage you to take a look at this blog post, which takes you through a self-hosted server on GCP; the methods should be nearly the same when deploying elsewhere (AWS, Digital Ocean, Azure etc).
a

Aurélien Vallée

04/14/2021, 6:29 AM
thanks, I'm going to read that and revert
n

nicholas

04/14/2021, 6:29 AM
It boils down to 2 ports, as you mentioned, the Apollo server port (which defaults to 4200), and the UI port (which defaults to 8080). So long as the other services can communicate with each other, which is true in the default Docker network, those should allow you to deploy to a remote server and access everything as you'd expect 🙂
Also, if you're more comfortable with Kubernetes, Prefect has a published helm chart to get you up and running with that!
a

Aurélien Vallée

04/14/2021, 6:32 AM
Not looking for kubernetes at the moment, just a simple local install
n

nicholas

04/14/2021, 6:33 AM
Then definitely the first one!
a

Aurélien Vallée

04/14/2021, 6:44 AM
hmm, read it, it clarified some things, but still, the exposed ports are listening on 0.0.0.0
basically I guess I'm looking for a settings allowing to tweak the published ports in the
docker-compose.yml
, so that I can expose e.g. 8080 specifically on
10.0.0.10
.
i.e.
-p 10.0.0.10:8080:8080
n

nicholas

04/14/2021, 6:52 AM
Gotcha. Try modifying your
~/.prefect/config.toml
like this:
[server]
host = "10.0.0.10"

    [server.database]
    host = "10.0.0.10"

    [server.graphql]
    host = "10.0.0.10"

    [server.hasura]
    host = "10.0.0.10"

    [server.ui]
    host = "10.0.0.10"
    apollo_url = "<http://10.0.0.10:4200/graphql>"
I can't test that at the moment but I'd be surprised if that didn't work
a

Aurélien Vallée

04/14/2021, 7:08 AM
hmmm, but since hasura and the db for instance are not published, i don't think that can work right?
i.e. i'm running the server with
prefect server start --no-hasura-port --no-postgres-port --no-graphql-port
so that only 4200 and 8080 are exposed on the host, and I want these 2 exposed ports to be listening on 10.0.0.10 instead if 0.0.0.0
i tried with:
backend = "server"

[server]
endpoint = "10.0.0.10:4200/graphql"

  [server.ui]
    host = "10.0.0.10"
    apollo_url = "<http://10.0.0.10:4200/graphql>"
but that's still not working, I see the ports listening on
0.0.0.0
# sudo lsof -i -P -n | grep LISTEN
[...]
docker-pr 268858            root    4u  IPv6 857627      0t0  TCP *:4200 (LISTEN)
docker-pr 269076            root    4u  IPv6 858070      0t0  TCP *:8080 (LISTEN)
by tweaking the published ports in the
docker-compose.yml
and using
docker-compose up
i can have these specific ports listening on
10.0.0.10
once exposed, but i'd like to work with
prefect server start
instead of
docker-compose
directly, since there's a bunch of things I cannot tweak, from working solely from the
docker-compose.yml
(and TBH it's more painful that working with
prefect server start
anyway)
I think what I need is the equivalent of
host_port
but for
host_bind_ip
Seems like the prefect agent is confused too:
$ cat ~/.prefect/config.toml 
backend = "server"

[server]
host = "10.0.0.10"

  [server.ui]
    host = "10.0.0.10"
    apollo_url = "<http://10.0.0.10:4200/graphql>"

$ echo $(curl -s 10.0.0.10:4200/graphql/)
GET query missing.

$ prefect agent local start
Traceback (most recent call last):
[...]
  File "/opt/.../virtualenv/prefect/lib/python3.8/site-packages/requests/sessions.py", line 742, in get_adapter
    raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for '10.0.0.10:4200'
Found that last issue, was because the
[server]
host
should have a valid scheme, i.e.
http://
https://docs.prefect.io/orchestration/server/telemetry.html recommends
[telemetry]
    [server.telemetry]
        enabled = false
while https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L50 uses:
[server]
    [server.telemetry]
    enabled = true
I guess the former from the doc is incorrect (?)
z

Zanie

04/14/2021, 2:12 PM
Hey @Aurélien Vallée -- glad it sounds like you've sorted out most of your issues. That block in the docs is indeed wrong but
toml
actually doesn't know anything about nesting indentation and the
[telementry]
block is meaningless there.
:upvote: 1
👍 1
a

Aurélien Vallée

04/14/2021, 3:17 PM
Pretty much all sorted out yes :-) still did not manage to choose which IP the published ports should bind to. I circumvent that with additional iptables rules but it's very fragile