Hi Prefect team, I’m evaluating Prefect for use at...
# prefect-server
Hi Prefect team, I’m evaluating Prefect for use at my company and had a few questions i couldn’t find answers in the docs: 1. Does anyone know of a k8s deployment option for prefect-server? All i can find online is the docker-compose option using
prefect server start
2. Is it possible to register an already-built (docker storage) flow? Our product involves deploying to many customer environments that are dynamic, so we would like to be able to build a flow image in CI and deploy it to our customer installations. The question is then, how do you register a flow in an arbitrary number of environments when the build step is before the register step? Does anyone know good patterns to follow here? Thanks!
re #1, i found this thread with some k8s resources: https://prefect-community.slack.com/archives/C014Z8DPDSR/p1597416524268000
Hi Ian, 1. There's currently no officially supported helm chart for prefect server. Some people have put up some rough versions, but no official version exists. A nice helm chart would be a welcome addition if you'd be interested in working on this (it's on our todo list, but probably will be a while without external contributors). 2. Yes. You have two options here: • Use a
storage, and manually specify the image name/tag/repository, then pass
when calling register. You'd be responsible for getting the flow code into the image and configuring the
storage with the proper flow
). • Store the flow externally in e.g. s3, and specify your image as part of the flow configuration. This lets you use a static image for multiple flows, as the flow code will be stored elsewhere.
Hi @Jim Crist-Harif thanks for the quick response. do you know if it’s possible to register a flow from a Docker storage container? typically with our customer installations, we ship a registry of containers, then use kubernetes to stand up the platform. If we could use the same containers to register the flow, it seems like we could create a Kubernetes Job to register them at install time?
Right now we wouldn’t have an easy way to run the Flow source code that is not containerized
If you're using non-docker storage, then yes (or using docker storage with a pre-built image and passing
), this is fairly straightforward, as no docker build step is required.
If you're using
storage and want to do the docker build step there, you'd need to use some kind of docker-in-docker setup. There's guides elsewhere for doing this, but I don't have anything immediate to point you to (sorry).
Usually we recommend users in this situation find some external storage that would work for them to store the flow code, so they don't need to do a docker build to store the flow code.
I think we would do the docker build step in our CICD process that packages all the containers into the installer. The question is then during the install, do we need another copy of the flow code to execute
? I am not familiar with what the docker build process stores in the flow container, but if the source code is there, it seems like we could run that?
something like
docker run <pre-built-flow-container> /path/to/flow/source/register.py
Yeah, that'd work. If the source code is available, you could also register with the code external to the containers.
ok awesome, thanks Jim!