I've setup a prefect server on a Google compute instance, but I still have problems grasping some concepts:
I have the server and local agent running on the GCE instance and the UI is working. I can register and run a flow that resides on the GCE machine without errors.
Now I want to setup GitLab as code repository. My initial intuition was: I can probably setup a connection to my Gitlab repo in a config file and then all flows will be synced from there. When I commit something, it will be updated in prefect as well.
This is not the case as it seems. I'm trying to wrap my head around how to use the Storage options, but I really don't understand it.
On my server, do I have to create python files for each of my flows that reside in Gitlab? What sense would the repository make then?
Thanks for your help! I'm really lost here...
01/27/2022, 4:33 PM
I think you are talking about a CI/CD from Gitlab to register the Flows. The Storage is just to denote where the Flow lives so that Prefect knows where to pull it from during execution time.
Your server does not host any compute or the files. You can have an independent repo that Prefect will pull the files from during execution time. The repo can also be an S3 bucket or a Docker repository. There just needs to be something for Prefect to pull the Flows from.
01/27/2022, 4:45 PM
Thanks Kevin! I think I understand it better now:
1. I create a file on my local computer and push it to my Gitlab repo.
2. On my local computer I register the flow via the remote server
3. When running a schedule, Prefect looks for the latest file in the repo.
This probably means, that I have to add the same Storage config to every flow file?
01/27/2022, 4:47 PM
It’s not the same for Gitlab because you have to point to the file path in that repo right?