Hi Prefect team! I have a couple questions about Orion capabilities within our production environment. Currently we are running Prefect core with prefect server on a local production server and agents on separate vm servers. This makes it really easy to run automated capabilities, process flows quickly, etc. We really like how flows can be defined and encapsulated in docker with dependencies and shared within our architecture. We don’t like how Flow of Flows is statically defined within Prefect Core, but I believe this will be solved with prefect Orion.
• Am I going to be able to set Orion up in a similar architecture? Can I have Orion set up on a local server that employees can access via web ui and run capabilities? Will it be able to store flow data and logs?
• I was wondering what docker storage is going to look like within a production infrastructure. Is there going to be support for docker storage in gitlab like with prefect core?
• I really like Orion’s developer simplicity and how easy it is to create flow of flows. Is there going to be a way for flow of flows to be dynamically created in the UI? As in, like some sort of functionality to drag and drop flows in a particular order or check box which flows you want to run.
Thanks for all your help!
03/24/2022, 2:42 PM
Hi @Nick Hart, yes you will be able to have the same server/agent setup. But also note that it’s much more trivial to spin up Orion as well so you can have multiple of them. Yes on the UI-flow data-logs part. Similar to Server, if you need auth, then you would go to Cloud 2.0
Someone may correct me about Docker storage, but I believe there will be support. More interesting though is that we will work on how to package user dependencies for them and this may eliminate the need for Docker in some cases.
GUI based flows….way out in the future. Open to that path, not an immediate priority
Also wanna note, that a lot of these may not be smooth at this very instant. Storage, for example, is being worked on now to make it more definitive.
03/24/2022, 2:49 PM
Thanks @Kevin Kho! We would be unable to use Cloud 2.0 as the production network is not connect to the internet. Would we be able to setup our own internal cloud 2.0?
And as long as we can package dependencies that should work.
GUI based flows would be cool!
And yes of course. i'm sure everyone is asking,, but do you have an idea of the timeline for releases?
03/24/2022, 2:51 PM
You actually will be able to have cloud on your own infrastructure! No public timeline. In the next quarter or so we should see more stability and production-readiness from Orion and Cloud
03/24/2022, 4:04 PM
@Nick Hart Cloud 2.0 uses the same hybrid execution model allowing you to have the entire execution layer deployed in your private infrastructure, but you will need to allow outbound traffic in order to communicate with Cloud.
In the future, we will explore the option of providing a completely dedicated infrastructure that might allow Cloud 2.0 deployments that are completely offline, but we can't make any guarantees at this point. Wanted to clarify to give you full transparency.
03/24/2022, 4:20 PM
@Anna Geller basically I am wondering with Orion, am I able to stand up a dedicated server that can act as a central hub allowing users to kick off flows (and maybe even build flow of flows) and have a separate computational cluster (most likely with dask)? Or some sort of infrastructure like this without any network activity outside of the local network
03/24/2022, 4:25 PM
Yeah so we chatted about this internally. The central hub with Orion is doable similar to Prefect Server. The full Cloud experience on-premise is the one that is to be explored.