Hello everybody. I was using the gitlab storage se...
# prefect-community
j
Hello everybody. I was using the gitlab storage setting in version 1. However, it seems that version 2 is not supported. Do you know how? If there is no way, when will the gitlab storage be updated?
1
a
We want to integrate with Gitlab to allow you to easily push your code to your repo and then trigger a CI/CD pipeline deploying your flow from there. So while it's not yet decided whether we may support that as storage, we will certainly provide integration with Gitlab, e.g. via CI/CD recipe
🙌 1
a
but what do you offer instead of gitlab storage not and in future?
a
The file system block docs list available storage options
a
DockerStorage is also off the table, right?
a
As a concept yes, but as a problem to be solved and as a feature, not at all. You should see a way of packaging both flow code and dependencies into a single image within 1-2 weeks
🎉 2
b
I saw that a GitHub storage block was introduced — particularly one that can be used for deployments. In the context of that, would that create precedent/space for the contribution of a GitLab block?
a
In theory yes but no plans to add that atm, GitHub is mainly meant for incremental adoption to get started easily, for production we still recommend remote storage blocks or packaging code into container image
👍 1
b
Incremental adoption makes sense. I have seen elsewhere where you’ve talked about the emphasis on production deployment done using other storage methods (I.e., we are using SMB - since my organization doesn’t provide us much access to cloud storage services - including GitHub). With regards to incremental adoption, I think GitLab makes sense in a similar way to GitHub, since it’s used by such a broad business user base (just less visible, since so much of it is on-premise and being firewalls). For that reason, and prior to seeing your response, I did experiment and create a proof of concept GitLab storage block based on the GitHub one that works (including authentication to a private repo). That said, I certainly have no issues shelving the idea. I definitely appreciate you humoring my less-common use case scenarios. EDIT — I also just noticed the newer Issue templates introduced a little while back. I can move these kinds of feature and enhancement questions there from now on, since it seems like that’s the intention with those, rather than bothering you here about it.
a
Thanks for explaining more. How are you using Prefect? if you are running your agent e.g. on an on-prem VM, then you don't need storage at all, you can use this local file system on the VM. Similarly, if you don't want to use remote storage blocks, I think the Docker image UX shown here might be a really good option https://medium.com/the-prefect-blog/prefect-2-3-0-adds-support-for-flows-defined-in-docker-images-and-github-repositories-79a8797a7371
not discouraging GitLab block but I'd love to persuade you that there are some potentially much nicer and robust ways to tackle the problem of "I want to run deployments without remote storage blocks"
🙌 1
b
Thanks for the response! My thought was actually more on the contribution level. We are currently using Prefect 1 with Docker for our run time environments. Most of our flows are pulled from an on-premise GitLab and run in shared images. A handful of complex flows are using dedicated images where the flow is baked into the image itself (only because Prefect 1 Git[Lab] storage won’t allow easy use of a submodules. I’m currently working on our move to Prefect 2, and for that we are continuing to use Docker as our runtime, but are moving to SMB storage for deployments (like I said, I like what you’ve said elsewhere about version control/CICD tools remaining version control/CICD tools). We can’t use cloud storage, so it was definitely beneficial that I was encouraged to contribute the SMB block to Prefect (whether or not you remember - you engaged with me about this on a Slack account associated with my work’s organization - trying to consolidate my Prefect community interaction away from multiple usernames🙂). My thought on the GitLab block was mainly on the contribution level itself, not for use as a primary storage option ourselves (though, options are nice, and there are instances where it might be useful for dev/testing/interim storage during migration, etc). I know that there are many organizations that use on-premise GitLab storage, especially organizations that value their code and data remaining on their own infrastructure to a strong degree (I.e., not even in Azure/AWS, etc), and wonder if GitLab being available with the same intention as GitHub might make sense (even if text in docs and/or Orion UI and/or terminal messages show up about it not being intended for full production development). But, like I said, if it doesn’t fit the vision for Prefect’s path forward, that’s ok too. Even if Prefect is good with the idea of a GitLab block for dev/testing/transition, would a more helpful path connected to Prefect’s intended path forward be to contribute other non-cloud storage blocks (like maybe SFTP or SCP, etc)?
🙏 1
a
would a more helpful path connected to Prefect’s intended path forward be to contribute other non-cloud storage blocks (like maybe SFTP or SCP, etc)?
Yes, 100%, you are spot on here -- SFTP is a storage block to which we could write to and read from; GitHub and GitLab are really no storage systems, those are version control and engineering collaboration platforms so SFTP would be a much nicer way of solving the problem for on-prem deployments. I'd definitely love to see a contribution for that if you would want to submit a PR
👍 1
b
Sounds good. I’ll plan on doing that.
🙏 1
🙌 1