This might be random, but I understand the use of Prefect for developing/orchestrating data pipelines for data analytics/reporting. Do people also use Prefect (or open source in general) for Enterprise application/information/system integration (essentially synchronizing information between two operational systems)? When I try to research this topic all I can find are advertisements from PaaS/SaaS companies trying to sell their all-in-one solutions.
04/21/2022, 5:51 PM
We absolutely support that. Usually, when performing data replication between enterprise applications and your data warehouse (or other databases), you may use EL tools such as Airbyte or Fivetran. We have some built-in tasks that can help you use those tools and orchestrate such data replication workflows with Prefect:
• Prefect-Airbyte integration recipe and Airbyte task docs
• Fivetran tasks docs
EAI is such a broad term - can you explain your use case a bit more? which systems are you trying to connect together? are you looking for just data replication between specific systems, and if so can you tell which ones?
06/07/2022, 6:14 PM
Well we have to sync some data between operational systems, like syncing products/batches/change controls between our SAP and Veeva Quality Management system. Or syncing SOP metadata between our Veeva Quality Management system and our manufacturing systems. It's mostly syncing master data elements between systems, so you don't have to manually maintain that data in each separate system.
Sorry for the very late reply, I don't use slack frequently yet.
06/08/2022, 12:33 PM
That seems like a great question to the Airbyte community: https://airbyte.com/community
they also have discourse https://discuss.airbyte.io/
especially given that Airbyte now also supports reverse ETL, they try to support data replication across systems in a many to many relationships, not only between OLTP -> OLAP for analytics as most ELT tools do, but their plan is to support any OLTP/OLAP/HTAP <-> OLTP/OLAP/HTAP - pretty much the same concept as EAI when looking at it pragmatically