Im doing this clean install on Ubuntu 18, python 3...
# prefect-community
j
Im doing this clean install on Ubuntu 18, python 3.6 , steps:
new user: prefect , added to sudo and docker, nothing special regarding environment.
docker was already installed, I did add docker-compose
stopped my running postgres service, so it doesnt have any port conflicts, however would love to use this one.
pip install prefect
---> This creates a .local in my prefect home, navigate to /bin and /lib , took a closer look, run
prefect backend server
---> creates a .prefect with a .toml file with setting : backend = "server" , so far so good I guess. Now comes the next step in the documentation, run
prefect server start
---> Starts popping warning regarding postgres
Copy code
WARNING: The PREFECT_SERVER_DB_CMD variable is not set. Defaulting to a blank string.
WARNING: The DB_CONNECTION_URL variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
Pulling postgres ... done
Pulling hasura   ... done
Pulling graphql  ... done
Pulling apollo   ... done
Pulling towel    ... done
Pulling ui       ... done
WARNING: The PREFECT_SERVER_DB_CMD variable is not set. Defaulting to a blank string.
WARNING: The DB_CONNECTION_URL variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1   ... done
Creating tmp_graphql_1  ... done
Creating tmp_towel_1    ... done
Creating tmp_apollo_1   ... done
Creating tmp_ui_1       ... done
Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_towel_1, tmp_apollo_1, tmp_ui_1
graphql_1   | bash: -c: line 0: syntax error near unexpected token `&&'
graphql_1   | bash: -c: line 0: ` && python src/prefect_server/services/graphql/server.py'
hasura_1    | {"type":"pg-client","timestamp":"2020-10-20T20:15:35.282+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1    | {"type":"pg-client","timestamp":"2020-10-20T20:15:35.282+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1    | {"type":"startup","timestamp":"2020-10-20T20:15:35.282+0000","level":"error","detail":{"kind":"db_migrate","info":{"internal":"could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n","path":"$","error":"connection error","code":"postgres-error"}}}
postgres_1  | Error: Database is uninitialized and superuser password is not specified.
postgres_1  |        You must specify POSTGRES_PASSWORD to a non-empty value for the
postgres_1  |        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
postgres_1  |
postgres_1  |        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
postgres_1  |        connections without a password. This is *not* recommended.
postgres_1  |
postgres_1  |        See PostgreSQL documentation about "trust":
postgres_1  |        <https://www.postgresql.org/docs/current/auth-trust.html>
So where to start from here, adjusting docker compose file or other way to configure the local env?
e
You can use a toml file for config options. By default this file is
~/.prefect/config.toml
, its path and name can be customized by the env var
PREFECT__USER_CONFIG_PATH
. Check the
config.toml
included in the prefect repo, as a default to start with, for most if not all options you can tinker with, : https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml Also checking the cli for the
server
subcommand and the
docker-compose.yml
helped me a lot when setting up a server: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/server.py https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml
j
Thanks, for pointing to the git,
config.toml
are these settings username and password used while running the docker-compose? Within the yml
service: image: "postgres:11"
? I get confused.
e
Yeah, docker-compose.yml uses envvars such as
POSTGRES_PASSWORD
for the postgres container. Therefore, when it is time for
prefect server
to start docker-compose, these environment variables must be present.
prefect server
subcommand temporarily sets env vars such as
POSTGRES_PASSWORD
with values read from the config file:
config.toml
. If a config file cannot be found, or a requested config parameter isn’t in the file, I believe it takes a
None
value. This means prefect sets
POSTGRES_PASSWORD
to an empty value. This I believe is why your postgres container had an empty password.
j
Thanks so far, but no luck, my exported env vars arent picked up, gonna add some code locally here ...
😞
It does load the config.toml
So in the code [336]:
cmd = ["docker-compose", "up"]
proc = subprocess.Popen(cmd, cwd=compose_dir_path, env=env)
it should have the env vars,
Found something regarding
docker-compose
... pretty annoying. https://prefect-community.slack.com/archives/CL09KU1K7/p1603235984359800