k

    Kyle

    1 year ago
    Hi all! I've got a weird issue when starting the prefect server. If anyone have ever faced this issue before, please help me.
    WARNING: The PREFECT_SERVER_DB_CMD variable is not set. Defaulting to a blank string.
    WARNING: The DB_CONNECTION_URL variable is not set. Defaulting to a blank string.
    WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
    WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
    WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
    Creating network "prefect-server" with the default driver
    Creating tmp_postgres_1 ... done
    Creating tmp_hasura_1   ... done
    Creating tmp_graphql_1  ... done
    Creating tmp_towel_1    ... done
    Creating tmp_apollo_1   ... done
    Creating tmp_ui_1       ... done
    Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_towel_1, tmp_apollo_1, tmp_ui_1
    graphql_1   | bash: -c: line 0: syntax error near unexpected token `&&'
    graphql_1   | bash: -c: line 0: ` && python src/prefect_server/services/graphql/server.py'
    hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
    hasura_1    | {"type":"pg-client","timestamp":"2020-10-26T11:45:30.504+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
    hasura_1    | {"type":"startup","timestamp":"2020-10-26T11:45:30.504+0000","level":"error","detail":{"kind":"db_migrate","info":{"internal":"could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n","path":"$","error":"connection error","code":"postgres-error"}}}
    ui_1        | πŸ‘ΎπŸ‘ΎπŸ‘Ύ UI running at localhost:8080 πŸ‘ΎπŸ‘ΎπŸ‘Ύ
    postgres_1  | Error: Database is uninitialized and superuser password is not specified.
    postgres_1  |        You must specify POSTGRES_PASSWORD to a non-empty value for the
    postgres_1  |        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
    postgres_1  | 
    postgres_1  |        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
    postgres_1  |        connections without a password. This is *not* recommended.
    postgres_1  | 
    postgres_1  |        See PostgreSQL documentation about "trust":
    postgres_1  |        <https://www.postgresql.org/docs/current/auth-trust.html>
    tmp_hasura_1 exited with code 1
    tmp_graphql_1 exited with code 1
    tmp_postgres_1 exited with code 1
    Billy McMonagle

    Billy McMonagle

    1 year ago
    Your postgres container is failing to initialize because the POSTGRES_PASSWORD variable is not set correctly.
    k

    Kyle

    1 year ago
    (venv) shark@shark-H310M-S2-2-0:~/Documents/prefect$ prefect config
    
    {'debug': False, 'home_dir': '/home/shark/.prefect', 'backend': 'cloud', 'server': {'host': '<http://localhost>', 'port': 4200, 'host_port': 4200, 'endpoint': '<http://localhost:4200>', 'database': {'host': 'localhost', 'port': 5432, 'host_port': 5432, 'name': 'prefect_server', 'username': 'postgres', 'password': 'postgres', 'connection_url': '<postgresql://postgres:postgres@localhost:5432/prefect_server>', 'volume_path': '/home/shark/.prefect/pg_data/'}, 'graphql': {'host': '0.0.0.0', 'port': 4201, 'host_port': 4201, 'debug': False, 'path': '/graphql/'}, 'hasura': {'host': 'localhost', 'port': 3000, 'host_port': 3000, 'admin_secret': '', 'claims_namespace': 'hasura-claims', 'graphql_url': '<http://localhost:3000/v1alpha1/graphql>', 'ws_url': '<ws://localhost:3000/v1alpha1/graphql>', 'execute_retry_seconds': 10}, 'ui': {'host': '<http://localhost>', 'port': 8080, 'host_port': 8080, 'endpoint': '<http://localhost:8080>', 'apollo_url': '<http://localhost:4200/graphql>'}, 'telemetry': {'enabled': True}}, 'cloud': {'api': '<https://api.prefect.io>', 'endpoint': '<https://api.prefect.io>', 'graphql': '<https://api.prefect.io/graphql>', 'use_local_secrets': True, 'heartbeat_interval': 30.0, 'check_cancellation_interval': 15.0, 'diagnostics': False, 'logging_heartbeat': 5, 'queue_interval': 30.0, 'agent': {'name': 'agent', 'labels': [], 'level': 'INFO', 'auth_token': '', 'agent_address': '', 'resource_manager': {'loop_interval': 60}}}, 'logging': {'level': 'INFO', 'format': '[%(asctime)s] %(levelname)s - %(name)s | %(message)s', 'log_attributes': [], 'datefmt': '%Y-%m-%d %H:%M:%S', 'log_to_cloud': False, 'extra_loggers': []}, 'flows': {'eager_edge_validation': False, 'run_on_schedule': True, 'checkpointing': False, 'defaults': {'storage': {'add_default_labels': True, 'default_class': 'prefect.environments.storage.Local'}}}, 'tasks': {'defaults': {'max_retries': 0, 'retry_delay': None, 'timeout': None}}, 'engine': {'executor': {'default_class': 'prefect.engine.executors.LocalExecutor', 'dask': {'address': '', 'cluster_class': 'distributed.deploy.local.LocalCluster'}}, 'flow_runner': {'default_class': 'prefect.engine.flow_runner.FlowRunner'}, 'task_runner': {'default_class': 'prefect.engine.task_runner.TaskRunner'}}}
    This is the config of the prefect. I'm not sure what is wrong in there.
    Actually, I did nothing, I just installed Preffect with Conda and set backend as server(
    prefect backend server
    ) and start the prefect server running this command:
    prefect server start
    Billy McMonagle

    Billy McMonagle

    1 year ago
    I was running into this error very frequently a couple of weeks ago, but it was because I was trying to build my own set of container definitions. Using
    prefect server start
    should mean that you're using the default
    docker-compose.yml
    , which lives here https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml
    My guess is that you have setup a local
    ~/.prefect/config.toml
    file that doesn't have a postgres password set
    k

    Kyle

    1 year ago
    I tried to make the toml file to fix the issue as well. This is the current config.toml file.
    [server]
            [server.database]
            username = "postgres"
            password = "postgres"
            volume_path = "~/.prefect/pg_data/"
    Is there something wrong, @Billy McMonagle?
    Billy McMonagle

    Billy McMonagle

    1 year ago
    I'm not sure. The problem is specific to the postgres container. you might want to also set a value for $POSTGRES_DB. In the .toml file, you can add the key
    name = "your_db_name
    (I'm only sharing my experience since I had a similar issue, I am not affiliated w Prefect πŸ™‚ )
    k

    Kyle

    1 year ago
    Yeah, thank you so much, @Billy McMonagle However, I don't think the problem is not associated with missing POSTGRES_DB in the config.toml file, you know.
    I think there must be another reason, you know.
    Billy McMonagle

    Billy McMonagle

    1 year ago
    why don't you do this... my guess is that maybe one of the other containers is making an assumption about your user/password values...
    I used these values with success, for local testing:
    parameter :postgres_user, type: :String, default: "prefect"
    parameter :postgres_password, type: :String, default: "test-password"
    parameter :postgres_db, type: :String, default: "prefect_server"
    k

    Kyle

    1 year ago
    If I don't set username and password for the postgres in the config.toml file, do you think it works then?
    I don't use my own docker-compose.yml file. I just use default compose file, you know
    Billy McMonagle

    Billy McMonagle

    1 year ago
    I don't think so because I think you need to set a default value somewhere
    you could try making a copy of the docker-compose file in your project directly and running it directly, without going through the
    prefect
    CLI command.
    k

    Kyle

    1 year ago
    well... I don't think it's the right way to fix the issue.
    Billy McMonagle

    Billy McMonagle

    1 year ago
    up to you. I don't plan to use docker-compose in production at all. Β―_(ツ)_/Β―
    k

    Kyle

    1 year ago
    prefect was already providing the default values for the postgres db conntion, actually.
    Billy McMonagle

    Billy McMonagle

    1 year ago
    does that mean it's working for you?
    k

    Kyle

    1 year ago
    as you can see the above prefect config in JSON, values of the Postgres DB connection fields seem to be correct ( they're not empty ). but the error is raised when the postgres container is created through docker.
    I already tried to add the POSTGRES_USER in the config.toml file like the bellow and run prefect server, but was not working.
    [server]
            [server.database]
            name = "prefect_server"
            username = "postgres"
            password = "postgres"
            volume_path = "~/.prefect/pg_data/"
    Billy McMonagle

    Billy McMonagle

    1 year ago
    you might want to do a
    docker system prune
    to cleanup your local system.
    bc you are using a local volume, i'm wondering if you just need to wipe the data and start a fresh one
    I also found it useful to run just the pg container directly, it helps to minimize the number of moving pieces. eg
    docker run --rm   --name pg-docker -e POSTGRES_PASSWORD=password -p 5432:5432 postgres
    k

    Kyle

    1 year ago
    Thanks, Billy. let me try to clean everything(
    docker system prune -a
    ) and run it again. although I already did it before adding the
    name
    property to the config.toml file.
    Billy McMonagle

    Billy McMonagle

    1 year ago
    good luck! this can be frustrating, I only know docker well enough to get myself in trouble sometimes.
    k

    Kyle

    1 year ago
    yeah, I understand.
    Actually, this strange docker issue gets me stick in the same place for a long time.
    yeah, running the postgres image directly works now!
    hmm... but, how can we always run the prefect server like that?
    Jesper van Dijke

    Jesper van Dijke

    1 year ago
    run it from .
    local/bin/prefect server start
    not directly in your home, it will then use the toml. Search and read my comments here from the last few days, encountered the same problem... docker-compose will probably need a
    .env
    file
    k

    Kyle

    1 year ago
    Thanks for your message, Jasper. Let me play with that and let you know.
    However, I can not find
    bin
    directory within
    .local
    .
    (venv) shark@shark-H310M-S2-2-0:~/Documents/prefect$ ~/.local/
    lib/   share/