https://prefect.io logo
Channels
Powered by
m

Matthew Blau

12/29/2020, 1:23 PM
Hello all, my company is happy with what I was able to do with Prefect and wants to move forth with deploying it, however, I was told that we would prefer for Prefect itself to be in a docker container when we do the deployment. I do see that their is a docker image on docker hub, however I cannot locate any docs on how to build the image. Can someone point me in the right direction?
a

Adam Roderick

12/29/2020, 1:27 PM
@Matthew Blau you have a couple of options here https://docs.prefect.io/orchestration/recipes/configuring_storage.html
m

Matthew Blau

12/29/2020, 1:30 PM
Hi @Adam Roderick I have taken a look at this and it appears that this is for configuring flows to use docker storage, is that correct? I am looking to install prefect itself in a docker container from the image provided here https://hub.docker.com/r/prefecthq/prefect.
a

Adam Roderick

12/29/2020, 2:04 PM
"however I cannot locate any docs on how to build the image" -- does this mean you are new to docker and want to know how to build an image? Or are you looking to customize the image?
m

Matthew Blau

12/29/2020, 2:05 PM
I am fairly new to docker, yes. The images that I've worked with up til now have had build instructions. I'm looking to build and test what I was able to do previously.
a

Adam Roderick

12/29/2020, 2:06 PM
That prefect image is pre-built. You don't need to build it, just run the docker pull command that is shown on the page. That will download the image to your local machine. Then you can start up a container from the image with the docker run command
m

Matthew Blau

12/29/2020, 2:07 PM
Ahh! That is what I was missing in my understanding. Thank you!
@Adam Roderick I appear to be something wrong as when I attempt the docker run command this is the output I see:
Copy code
mblau@mblau-VirtualBox:~/docker_prefect$ docker -it run prefecthq/prefect:latest 
tini (tini version 0.18.0)
Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version

Execute a program under the supervision of a valid init process (tini)

Command line options:

  --version: Show version and exit.
  -h: Show this help message and exit.
  -s: Register as a process subreaper (requires Linux >= 3.4).
  -p SIGNAL: Trigger SIGNAL when parent dies, e.g. "-p SIGKILL".
  -v: Generate more verbose output. Repeat up to 3 times.
  -w: Print a warning when processes are getting reaped.
  -g: Send signals to the child's process group.
  -e EXIT_CODE: Remap EXIT_CODE (from 0 to 255) to 0.
  -l: Show license and exit.

Environment variables:

  TINI_SUBREAPER: Register as a process subreaper (requires Linux >= 3.4).
  TINI_VERBOSITY: Set the verbosity level (default: 1).
  TINI_KILL_PROCESS_GROUP: Send signals to the child's process group.
Could you advise?
j

Jim Crist-Harif

12/29/2020, 3:01 PM
Ah, the image doesn't set a command by default, you'll want to do something like
Copy code
docker run -it --rm prefecthq/prefect:latest /usr/bin/env bash
we'll want to update those docs, thanks for pointing that out.
m

Matthew Blau

12/29/2020, 3:16 PM
@Jim Crist-Harif ah! Thank you. I have ran that command as you put it up there and it has dropped me into a root shell of the container. Is there anything else I need to do to get the UI up and running? My use case is to run other docker containers and I was successful in doing that with the install not in docker. Anything I should be aware of that needs handled in order for me to be able to talk to the other containers?
j

Jim Crist-Harif

12/29/2020, 3:19 PM
Ah, sorry, it wasn't clear to me what you were trying to do here. That just got you into an active session inside a prefect docker image. What is your goal here? Are you trying to run Prefect Server? Or build Prefect Flows as docker containers? Or?
m

Matthew Blau

12/29/2020, 3:22 PM
I'm attempting to run Prefect Server so I can run existing docker containers. My company builds out docker containers for all our integrations and I am wanting to quickly move them over to being handled by prefect for orchestration. So far I've pulled down the docker image and want to get the Server Web UI up and running
j

Jim Crist-Harif

12/29/2020, 3:23 PM
Ah, gotcha. For that, you should run 
prefect server start
(if you have docker running locally) and everything should just work. See https://docs.prefect.io/orchestration/server/deploy-local.html
The 
prefecthq/prefect
image is for running flows, not prefect server (that's a separate set of images).
m

Matthew Blau

12/29/2020, 3:27 PM
@Jim Crist-Harif ah so then the supported method of installing the server is via pip or a virtual environment? I had already done pip install prefect and wanted to use the prefect docker image as my boss indicated he would like it to be dockerized for running in production. But if that's not a supported method of running it I can likely convince him of using a virtual environment instead
j

Jim Crist-Harif

12/29/2020, 3:29 PM
Prefect Server is composed of multiple services. We provide a 
docker-compose
backed setup for deploying it as a set of docker containers by running 
prefect server start
. This requires a local install of 
prefect
to run the initial command, but the actual server services run inside several docker containers in your environment.
m

Matthew Blau

12/29/2020, 3:32 PM
Ahh so I should just install it via pipenv install --pre prefect
j

Jim Crist-Harif

12/29/2020, 3:33 PM
Sure, that'd work.
m

Matthew Blau

12/29/2020, 3:42 PM
@Jim Crist-Harif awesome! I shall advise my boss that is the route we will be taking.
@Jim Crist-Harif One last question for now, if I can bother you some more, I had noticed that in my testing of starting and stopping the server projects that I create do not always persist. Is this the relevant bit of documentation that I should be referencing to ensure that things like that will persist in the event of the server going offline for any reason? https://docs.prefect.io/core/concepts/persistence.html
j

Jim Crist-Harif

12/29/2020, 3:49 PM
If you're starting and stopping Prefect Server, you'll need to persist the database somewhere. See https://docs.prefect.io/orchestration/server/deploy-local.html#database-persistence-and-migrations for how to do this in a dockerized setup.
The docs you linked are for persisting data at the flow-run-level (which is different than the data that prefect server persists)
m

Matthew Blau

12/29/2020, 3:52 PM
Ah awesome! Thank you again. I am getting the hang of your documentation though I do get lost sometimes in it. We are looking to deploy this corp wide in the coming days so I want to make sure I take all the necessary steps to configure it in a way that is the best and sanest way of setting up
j

Jim Crist-Harif

12/29/2020, 3:54 PM
Great! We also have a new helm chart if a kubernetes based deployment would make more sense for you (and there's always Prefect Cloud for our hosted offering). Lots of ways to get a production setup going.
m

Matthew Blau

12/29/2020, 6:42 PM
@Jim Crist-Harif one more question, if you don't mind. As we will be deploying this to production shortly and
Copy code
prefect server start
has a whole bunch of logging information in the terminal, is there a best practices deployment strategy that allows for us to hide that away like we would pipe other things to dev/null so that it can continue to run without being in the foreground? Any docs that you can point me to?
j

Jim Crist-Harif

12/29/2020, 6:43 PM
We just added an option to start all services in the background.
Copy code
prefect server start -d
should start all your services as docker containers, then return with them running in the background. You can do
Copy code
prefect server stop
later to stop everything. This requires prefect 0.14.1, which was just released today.
m

Matthew Blau

12/29/2020, 6:49 PM
Oh perfect! so when I do the pipenv install of Prefect it will pull down and install the latest version, correct? and can I chain flags together like :
Copy code
prefect server start -d --use-volume
so I can configure persistence as well?
j

Jim Crist-Harif

12/29/2020, 6:51 PM
it will pull down and install the latest version, correct?
I'm not familiar with pipenv, but most tools for installing python libraries install the latest versions by default. You can check your version of prefect with 
prefect version
. And yes, the above command should work fine.
m

Matthew Blau

12/29/2020, 7:45 PM
@Jim Crist-Harif btw https://docs.prefect.io/orchestration/server/deploy.html#database-persistence-and-migrations 404's. I was trying to click on "persistent volume"
j

Jim Crist-Harif

12/29/2020, 7:46 PM
Thanks! I'll put a fix up now.
That should link to the heading above that section, FYI
Powered by