https://prefect.io logo
Powered by
Title
k

Kyle

10/26/2020, 11:47 AM
Hi all! I've got a weird issue when starting the prefect server. If anyone have ever faced this issue before, please help me. 
WARNING: The PREFECT_SERVER_DB_CMD variable is not set. Defaulting to a blank string.
WARNING: The DB_CONNECTION_URL variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1   ... done
Creating tmp_graphql_1  ... done
Creating tmp_towel_1    ... done
Creating tmp_apollo_1   ... done
Creating tmp_ui_1       ... done
Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_towel_1, tmp_apollo_1, tmp_ui_1
graphql_1   | bash: -c: line 0: syntax error near unexpected token `&&'
graphql_1   | bash: -c: line 0: ` && python src/prefect_server/services/graphql/server.py'
hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1    | {"type":"startup","timestamp":"2020-10-26T11:45:30.504+0000","level":"error","detail":{"kind":"db_migrate","info":{"internal":"could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n","path":"$","error":"connection error","code":"postgres-error"}}}
ui_1        | πŸ‘ΎπŸ‘ΎπŸ‘Ύ UI running at localhost:8080 πŸ‘ΎπŸ‘ΎπŸ‘Ύ
postgres_1  | Error: Database is uninitialized and superuser password is not specified.
postgres_1  |        You must specify POSTGRES_PASSWORD to a non-empty value for the
postgres_1  |        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
postgres_1  | 
postgres_1  |        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
postgres_1  |        connections without a password. This is *not* recommended.
postgres_1  | 
postgres_1  |        See PostgreSQL documentation about "trust":
postgres_1  |        <https://www.postgresql.org/docs/current/auth-trust.html>
tmp_hasura_1 exited with code 1
tmp_graphql_1 exited with code 1
tmp_postgres_1 exited with code 1
b

Billy McMonagle

10/26/2020, 1:16 PM
Your postgres container is failing to initialize because the POSTGRES_PASSWORD variable is not set correctly.
k

Kyle

10/26/2020, 1:23 PM
(venv) shark@shark-H310M-S2-2-0:~/Documents/prefect$ prefect config

{'debug': False, 'home_dir': '/home/shark/.prefect', 'backend': 'cloud', 'server': {'host': '<http://localhost>', 'port': 4200, 'host_port': 4200, 'endpoint': '<http://localhost:4200>', 'database': {'host': 'localhost', 'port': 5432, 'host_port': 5432, 'name': 'prefect_server', 'username': 'postgres', 'password': 'postgres', 'connection_url': '<postgresql://postgres:postgres@localhost:5432/prefect_server>', 'volume_path': '/home/shark/.prefect/pg_data/'}, 'graphql': {'host': '0.0.0.0', 'port': 4201, 'host_port': 4201, 'debug': False, 'path': '/graphql/'}, 'hasura': {'host': 'localhost', 'port': 3000, 'host_port': 3000, 'admin_secret': '', 'claims_namespace': 'hasura-claims', 'graphql_url': '<http://localhost:3000/v1alpha1/graphql>', 'ws_url': '<ws://localhost:3000/v1alpha1/graphql>', 'execute_retry_seconds': 10}, 'ui': {'host': '<http://localhost>', 'port': 8080, 'host_port': 8080, 'endpoint': '<http://localhost:8080>', 'apollo_url': '<http://localhost:4200/graphql>'}, 'telemetry': {'enabled': True}}, 'cloud': {'api': '<https://api.prefect.io>', 'endpoint': '<https://api.prefect.io>', 'graphql': '<https://api.prefect.io/graphql>', 'use_local_secrets': True, 'heartbeat_interval': 30.0, 'check_cancellation_interval': 15.0, 'diagnostics': False, 'logging_heartbeat': 5, 'queue_interval': 30.0, 'agent': {'name': 'agent', 'labels': [], 'level': 'INFO', 'auth_token': '', 'agent_address': '', 'resource_manager': {'loop_interval': 60}}}, 'logging': {'level': 'INFO', 'format': '[%(asctime)s] %(levelname)s - %(name)s | %(message)s', 'log_attributes': [], 'datefmt': '%Y-%m-%d %H:%M:%S', 'log_to_cloud': False, 'extra_loggers': []}, 'flows': {'eager_edge_validation': False, 'run_on_schedule': True, 'checkpointing': False, 'defaults': {'storage': {'add_default_labels': True, 'default_class': 'prefect.environments.storage.Local'}}}, 'tasks': {'defaults': {'max_retries': 0, 'retry_delay': None, 'timeout': None}}, 'engine': {'executor': {'default_class': 'prefect.engine.executors.LocalExecutor', 'dask': {'address': '', 'cluster_class': 'distributed.deploy.local.LocalCluster'}}, 'flow_runner': {'default_class': 'prefect.engine.flow_runner.FlowRunner'}, 'task_runner': {'default_class': 'prefect.engine.task_runner.TaskRunner'}}}
This is the config of the prefect. I'm not sure what is wrong in there.
Actually, I did nothing, I just installed Preffect with Conda and set backend as server(
prefect backend server
) and start the prefect server running this command: 
prefect server start
b

Billy McMonagle

10/26/2020, 1:59 PM
I was running into this error very frequently a couple of weeks ago, but it was because I was trying to build my own set of container definitions. Using 
prefect server start
should mean that you're using the default 
docker-compose.yml
, which lives here https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml
My guess is that you have setup a local 
~/.prefect/config.toml
file that doesn't have a postgres password set
k

Kyle

10/26/2020, 4:27 PM
I tried to make the toml file to fix the issue as well. This is the current config.toml file. 
[server]
        [server.database]
        username = "postgres"
        password = "postgres"
        volume_path = "~/.prefect/pg_data/"
Is there something wrong, @Billy McMonagle?
b

Billy McMonagle

10/26/2020, 5:03 PM
I'm not sure. The problem is specific to the postgres container. you might want to also set a value for $POSTGRES_DB. In the .toml file, you can add the key 
name = "your_db_name
(I'm only sharing my experience since I had a similar issue, I am not affiliated w Prefect πŸ™‚ )
k

Kyle

10/26/2020, 5:05 PM
Yeah, thank you so much, @Billy McMonagle However, I don't think the problem is not associated with missing POSTGRES_DB in the config.toml file, you know.
I think there must be another reason, you know.
b

Billy McMonagle

10/26/2020, 5:08 PM
why don't you do this... my guess is that maybe one of the other containers is making an assumption about your user/password values...
I used these values with success, for local testing: 
parameter :postgres_user, type: :String, default: "prefect"
parameter :postgres_password, type: :String, default: "test-password"
parameter :postgres_db, type: :String, default: "prefect_server"
k

Kyle

10/26/2020, 5:10 PM
If I don't set username and password for the postgres in the config.toml file, do you think it works then?
I don't use my own docker-compose.yml file. I just use default compose file, you know
b

Billy McMonagle

10/26/2020, 5:11 PM
I don't think so because I think you need to set a default value somewhere
you could try making a copy of the docker-compose file in your project directly and running it directly, without going through the 
prefect
CLI command.
k

Kyle

10/26/2020, 5:12 PM
well... I don't think it's the right way to fix the issue.
b

Billy McMonagle

10/26/2020, 5:13 PM
up to you. I don't plan to use docker-compose in production at all. Β―\_(ツ)_/Β―
πŸ‘ 1
k

Kyle

10/26/2020, 5:14 PM
prefect was already providing the default values for the postgres db conntion, actually.
b

Billy McMonagle

10/26/2020, 5:21 PM
does that mean it's working for you?
k

Kyle

10/26/2020, 5:23 PM
as you can see the above prefect config in JSON, values of the Postgres DB connection fields seem to be correct ( they're not empty ). but the error is raised when the postgres container is created through docker.
I already tried to add the POSTGRES_USER in the config.toml file like the bellow and run prefect server, but was not working. 
[server]
        [server.database]
        name = "prefect_server"
        username = "postgres"
        password = "postgres"
        volume_path = "~/.prefect/pg_data/"
b

Billy McMonagle

10/26/2020, 5:25 PM
you might want to do a 
docker system prune
to cleanup your local system.
bc you are using a local volume, i'm wondering if you just need to wipe the data and start a fresh one
I also found it useful to run just the pg container directly, it helps to minimize the number of moving pieces. eg 
docker run --rm   --name pg-docker -e POSTGRES_PASSWORD=password -p 5432:5432 postgres
k

Kyle

10/26/2020, 5:31 PM
Thanks, Billy. let me try to clean everything(
docker system prune -a
) and run it again. although I already did it before adding the 
name
property to the config.toml file.
b

Billy McMonagle

10/26/2020, 5:32 PM
good luck! this can be frustrating, I only know docker well enough to get myself in trouble sometimes.
k

Kyle

10/26/2020, 5:34 PM
yeah, I understand.
Actually, this strange docker issue gets me stick in the same place for a long time.
yeah, running the postgres image directly works now!
πŸŽ‰ 1
hmm... but, how can we always run the prefect server like that?
j

Jesper van Dijke

10/27/2020, 12:19 AM
run it from .
local/bin/prefect server start
not directly in your home, it will then use the toml. Search and read my comments here from the last few days, encountered the same problem... docker-compose will probably need a 
.env
file
k

Kyle

10/27/2020, 12:24 AM
Thanks for your message, Jasper. Let me play with that and let you know.
However, I can not find 
bin
directory within 
.local
.
(venv) shark@shark-H310M-S2-2-0:~/Documents/prefect$ ~/.local/
lib/   share/
#prefect-serverJoin Slack
Powered by