Good morning prefect community 😊. We are evaluating a switch to either Prefect or Dagster. On the surface the feature sets look quite similar. Do you have any resources aiding with the decision which solution to choose?

Welcome to the community @Julian Brendel, great to have you with us! Choosing the right tool is definitely important and something you would need to evaluate based on your use case. My recommendation would be, not surprisingly, to go with Prefect (especially Prefect 2.0) because it's an extremely versatile general-purpose orchestrator capable of addressing more use cases than Dagster, which is TTBOMK more focused on data warehousing (move data from A to B and transform it either along the way or after the load to the data warehouse). Can you say more about your use case? What would you like to orchestrate and operationalize/productionize using Prefect?

Hi Anna, Thank you for the swift response. Several years ago we have written our own workflow orchestration engine which we are now considering to replace with one of the amazing open source projects. Main consideration for us is simplicity of writing flows (prefect coming out as the clear winner here). Personally a big fan of prefect 1.0, however we are bit worried about the strong coupling between the server and client in 2.0. We are running a large amount of business critical jobs every day, concerned about Orion downtime shutting down ability to run Workflows. Is there guidance on running Orion as primary, however falling back to ephemeral in case of downtime? Currently we are using an external enterprise platform for scheduling which we are not ready to move away from, therefor would look to use prefect for flow orchestration, but not scheduling. Additionally we use oracle company wide, could you provide any info on potential support or guidance for writing our own adaptor. Sorry for the long post, Very much looking forward to exploring prefect further

Great question! I can understand why this may be a concern. With Cloud 2.0 you wouldn't have to worry about that since it runs in a highly available setting and is entirely managed for you. And while it's a managed SaaS product, your code and data remain private due to the Hybrid Execution Model. However, if you want to self-host and use purely the OSS product, it's possible to run Orion in a highly available fashion and we may provide more guidance about that in the second half of the year. The coupling between client and server shouldn't be a concern once you have more than one server and are able to load balance. For Oracle - we are working on providing an abstraction that will allow you to securely store e.g. DB credentials - is this what you are looking for? Or do you need a utility for writing queries from Python/Prefect?

Thank you very much, looking forward to further guidance on load balancing Orion. We are looking to use Oracle as backend for Orion instead of Postgres

I don't think that using Oracle as Orion backend will be possible - we support SQLite and Postgres. You could interact with an Oracle database in your flows but using Oracle as the Orion backend won't work

Is that something that could be requested for the backlog? Otherwise any advise on writing our own DB adaptor to support Oracle would be highly appreciated.

Why is this important to you? We can't support Oracle as Prefect 2.0 backend even if we wanted to - Oracle RDBMS is a commercial product, while Prefect 2.0 is an open-source product. You can interact with Oracle from your flows, but you can't use it as the orchestration backend - we provide an option to fully manage the orchestration backend for you with Cloud 2.0 - you can try it out and use it for a couple of months (free of charge) to get a feel for how it works

Hmm, unfortunately we are not quite ready to move into Prefect cloud as it's too big of a step from our current infrastructure. Investing into Orion would be a first step to ease future discussions regarding cloud. Oracle remains a standard for enterprise, all our DBA expertise and infrastructure lies with oracle. At the time being, having the discussion to move to prefect + invest in a new database solution will complicate things. Could you provide any guidance regarding writing our own Adapter, similar to the implentations done for PostgresSQL and SQLite?

Prefect 2.0 backend is built based on REST API services (built with FastAPI) communicating requests between a client and Orion server (uvicorn) talking to your backend database with the help of SQLAlchemy ORM. To use a different database backend, you'd need to adjust the ORM code to use Oracle instead