Hi, I played around some more with prefect, but I could no wrap my mind around how to transition from our current system. Currently: • multiple docker containers separated by concerns/topics • (too many) cron jobs Goal: • multiple docker images • central orchestration Migration: 1. add prefect to project dependencies 2. rewrite cron as flow(s) 3. build image (automated) 4. register jobs (automated) I can not get a clear understanding of how to accomplish 3 and 4. We definitely also need an image to spin up in other environments. Do I build the image normally push it in the registry and use it in the run_config? If so how do I register the flow and with what storage config? I assume I will run register from within the container. But what is the storage config in that case? Any help would be appreciated. I will try to write about our transition once I figured it out. I believe we are not the only ones with a setup like this.

Hey @Florian Kühnlenz -- it may help to take a look at https://github.com/PrefectHQ/prefect/discussions/4042

Hi @Zanie , thanks for the link! But S3 or Github are not an option. I would really prefer if everthing would be contained in the same container or several if necessary.

Then you can use

DockerStorage

and prefect will build the image at registration time and package your flow into it

Okay, so I would build my image as normal, and then let prefect build another for the flow?

I think you may want to set the base image for the flow's DockerStorage to your image in that case

I tried to explain best I could. We have a bunch of containers running cron jobs. All with different dependencies. Now we want to migrate to prefect.

If your CI doesn't support building a docker image then I'm not sure you can avoid using Github/S3/something as a storage for your serialized flow. You can use Local storage if you don't want to rely on an external service but then you'll need to copy the serialized flow files to your execution environment.

No you misunderstood. The CI can build docker images for sure. But I can not install prefect within the CI environment, except inside the image I am building. So the only way to run prefect is inside the image. If prefect then tries to build, I assume it needs to build the image inside the image, right?

Ah sorry about that. Yes, you are correct. For this reason, we don't recommend trying to use DockerStorage if you cannot install/run prefect outside a Docker container. You could work around this by adding the flow manually to your docker image but we don't have a guide written for this because it's not a typical pattern. I'll see if I can get a description of what you'd have to configure.

That would be great! I feel like the situation we are in is not that untypical if you have to keep things on premise.

Hey! I ran into a bit of hiccup with this but it seems sorted 🙂 Here's the flow file test-docker-storage.py

from prefect import Flow, task
from prefect.storage import Local
from prefect.run_configs import DockerRun

@task(log_stdout=True)
def say_hello():
    print("Hello world")

with Flow("docker-storage-example") as flow:
    say_hello()

flow.storage = Local(stored_as_script=True, path="/flow.py", add_default_labels=False)
flow.run_config = DockerRun(image="image-with-flow:latest")

if __name__ == "__main__":
    flow.register("default")

Here's the minimal Dockerfile

FROM prefecthq/prefect:latest

ADD test-docker-storage.py /flow.py

Many thanks! I was wondering if the local storage is the way to go. I might only be able to try it out on Monday, but I will report back.

@Marvin archive "Storing and running flows in

Docker

without building

DockerStorage

at registration time"