So to continue with my thoughts on flow registration (https://prefect-community.slack.com/archives/CL09KU1K7/p1602793993098800). I received several requests to share my current setup. (cc @Billy McMonagle, @Dylan). This is mainly a POC of prefect as workflow orchestration engine. 1. I have a vagrant VM where i install docker and docker-compose. 2. I took https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml and modified it a bit. I also pushed all variables defined in this docker compose in .env file. BTW i was wondering why

prefect server start

does not let you to detach?? It has so much control on how whole prefect server is deployed yet you cant use on anything rather then local console. Also i found that there was no actual documentation on all services of prefect server and how to connect them to each other. I was able to understand it only by reverse-engineering docker compose file and prefect server cli source code 😔 Other than that there is no really fancy stuff here, just regular compose file with some variables defined. 3. There is a big security hole in the whole setup called apollo. UI (read: your web browser) need access to apollo endpoint. There is no auth in both UI and apollo, which means you have to put the whole service behind the VPN, at least. 4. Then i have this nifty little dockerfile wich defines flow execution and registration environment. Nothing fancy too, except you have to somehow forward an apollo endpoint address. It also uses the same image as used by docker agent as its base. 5. On every commit i run this code:

docker build -t hs-databricks:build_runtime -f flows/Dockerfile --build-arg APOLLO_PUBLIC_URL=$APOLLO_PUBLIC_URL .
# Register all flows
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e APOLLO_PUBLIC_URL=$APOLLO_PUBLIC_URL hs-databricks:build_runtime \
 python /src/hs-databricks/flows/registry.py

6. And there is a central flow registration script, which scans ./flows folded for python files, extracts flows from them, builds docker storage and registers the flows. It has no comments, but i wrote it just yesterday ☺️ My plans are, since this all is running in docker, try to deploy the prefect server + agent + flow storage in AWS Fargate, so i dont need to maintain any EC2 at all. Hope this will help anyone 👍

I did something similar. 👍 The only difference is not only did I setup the storage of all of the flows, but also, assign the proper

environment

to each flow. I originally used the same environment for all flows but have since iterated and made an internal library that translates placeholder objects (non-functional environments) with the production environments at registration. (These "dummy environments" are used to specify if the flow needs more or less resources or other configurations)

This is very useful and helps clarify a few Prefect concepts for me. In particular, I was not understanding what all of the fuss about "storage" as a concept was - but I think I see here that storage is how the flow executor knows where to read the actual flow code from - right? It definitely makes sense to have this as some part of your build process to ensure the latest versions of all your flows have been registered.

This was a huge selling point for P to me ... the fact that "where the flow lives" is not a mere afterthought 👏

Hello, I really liked your registry.py script, however, I was curious how you implemented your flow files, could you give me an example?