https://prefect.io logo
Title
s

Sylvain Hazard

03/21/2022, 2:38 PM
[Prefect 2.0][Orion] Running through the documentation and got a couple questions. Those are about production-related patterns so please tell me if it is a bit early for these discussions. ā€¢ If I understand what I'm reading correctly, Orion's
Storage
is quite different from Prefect 1.0's
Storage
as it does not only store flow code but also results. Does this mean that
GitStorage
is not coming back and the new recommended process is to deploy from the CI on validated changes ? Also, this is quite specific to my use case but having to connect to a cloud storage resource to run Orion on Kubernetes could be an issue. We are currently running with
GitStorage
and a Kubernetes-hosted PostgreSQL instance which means we can manage every Prefect component from Kubernetes configuration. ā€¢ The current version of the
KubernetesFlowRunner
seems to use a fixed specification for the created job. Reading the code, I could not find a way to specify a template the way 1.0's
KubernetesRun
allows us to. Being able to create precisely sized pods to run flows has been really useful for us and I'd love to be able to keep doing it. Is it a planned feature for the future ? :thank-you: !
a

Anna Geller

03/21/2022, 2:56 PM
Those are all fantastic observations, thanks for looking into that! We are aware of the difference between read-only storage classes (such as all Git-based storage classes) vs. read-and-write storage classes such as all object storage classes. We are currently working on that so I'd say we should postpone discussing this for now and you can follow šŸ”” our release-notes to stay updated on that. There will be definitely a way of using GitHub for storage, but hard to say what exactly this will look like for now. Regarding the Kubernetes job template, let me open an issue for that. @Marvin open "Orion: allow specifying a custom job template on a KubernetesFlowRunner"
k

Kevin Kho

03/21/2022, 3:08 PM
I believe the Kubernetes template similar to 1.0 is on the roadmap. We are just looking if there is a more elegant way to do it
s

Sylvain Hazard

03/21/2022, 3:21 PM
Thanks for the answers ! Pretty excited for those !
šŸ‘ 1