https://prefect.io logo
Title
m

Minakshi

04/17/2021, 1:48 AM
Whats the process to start the prefect server using a mounted volume? are there ore details on how to use the --use-volume command? How to setup a sample postgres mounted volume etc?
I created a postgres volume using
docker run --rm  --name pg-docker -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres
but erroring out while using
prefect server start --use-volume --volume-path $HOME/docker/volumes/postgres:/var/lib/postgresql/data
ERROR: for cli_postgres_1 Cannot create container for service postgres: invalid mode: /var/lib/postgresql/data
ERROR: for postgres Cannot create container for service postgres: invalid mode: /var/lib/postgresql/data
ERROR: Encountered errors while bringing up the project.
I also tried creating a common volume using
docker volume create pgdata
Then spinning up postgres with
docker run --rm  --name pg-docker -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 -v pgdata postgres
and then starting prefect server with
prefect server start --use-volume --volume-path pgdata
Got this exception
Named volume "pgdata:/var/lib/postgresql/data:rw" is used in service "postgres" but no declaration was found in the volumes section.
Exception caught; killing services (press ctrl-C to force)
ERROR: Named volume "pgdata:/var/lib/postgresql/data:rw" is used in service "postgres" but no declaration was found in the volumes section.
Traceback (most recent call last):
 
File "/Users/mkorad/.pyenv/versions/3.8.5/lib/python3.8/site-packages/prefect/cli/server.py", line 324, in start
  
subprocess.check_call(
 
File "/Users/mkorad/.pyenv/versions/3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
  
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
z

Zanie

04/19/2021, 7:41 PM
Hey @Minakshi -- if you do
--use-volume
we will create the volume for you (via docker-compose) and it defaults to
'~/.prefect/pg_data'
-- you can provide a custom path to the data using
--volume-path
The path set by
--volume-path
is used in the
docker-compose
file to map it into the container https://github.com/PrefectHQ/prefect/blob/e751130efcd1f44bd9eef4d5a58670beeace00aa/src/prefect/cli/docker-compose.yml#L18
m

Minakshi

04/19/2021, 10:09 PM
Thanks Michael, if i use the the default volume
'~/.prefect/pg_data'
wouldn't it be overwritten during server restarts? Also trying it out :)
z

Zanie

04/19/2021, 10:10 PM
I don't think so!
🙏 1
m

Minakshi

04/19/2021, 10:25 PM
great was able to see the state saved by use --use-volume Thanks!
Just a clarifying question, we have situations where our box is replaced during deployments, in that case can the right way to use volumes would be
prefect server start --volume-path=<<s3-path>>
Can I use an external path like s3? Does the above command look right for using a custom volume path?
z

Zanie

04/19/2021, 10:41 PM
I don't think that docker-compose supports S3 paths
You could sync the S3 path to your machine on startup / teardown or mount the S3 bucket to your local file system
Or use something like EFS
m

Minakshi

04/19/2021, 11:57 PM
thank you!