Hello Prefect community, I was curious to get people's opinion about Fivetran vs Airbyte vs Stitch or any other connectors. We want to be able to pull data from Salesforce, DWH (Looker, Snowflake, BQ, Redshift), Pendo, Heap and dump into BQ (our DWH). Read a bunch of blog posts, but was hoping to hear more personal experiences.

A very quick summary from my own trials: Fivetran is a rolls-royce. It works well, it is very comfortable, but is very expensive. Stitch, airbyte and Meltano (my tool of choice) all use open-source singer connectors, and so the main issue is often how well the open-source taps and targets work for you. A lot of minutiae suddenly surfaces. I haven’t used airbyte for a while because I don’t want to run my own web server for the UI, but I do feel like it is UI-first. Meltano can be defined in code and I have been able to stuff it into Prefect. But you probably need some programming competence with those solutions. There are a bunch of other options as well, without recommending anything in particular there is Rivery, Matillion data loader (not to be confused with the full matillion platform), Skyvia +++ That was kinda general, feel free to ask more. Oh, and keep the API in front of mind when looking at the SaaS options.

So primarily what I want are three things: 1. Connector Coverage, Fivetran for e.g. does not have BQ/Snowflake as a source but as a destination. 2. API access so that we can kick-start sync, connections programmatically 3. White label UI app/library, so that within our app users directly use Fivetran/Airbyte's UI to build the source object. Destination is us which we can pre-populate. I had been looking at Airbyte and it is lacking in all 3 with promise of 2 and 3 coming next year. Fivetran does 2 and 3 but not 1.

Yeah the bigquery/snowflake sources are hard to find. Who would ever want to move data out of there? 😝 But there are singer taps for them. I am able to do it with Meltano, and so since Airbyte is also based on Singer it shouldn’t be much of an issue for them to adapt it: https://hub.meltano.com/extractors/tap-bigquery https://hub.meltano.com/extractors/tap-snowflake The white-label app thing is new to me, are you building something customer-facing?

We are building an Enterprise app and I want to ingest data in real-time. We have built out real time infrastructure, but effectively need connectors ui and api to ensure a smooth self service experience.

How real-time is real-time for you? Even CDC tools I’m using does micro-batching. AFAIK, only Fivetran has “realtime” as an option.

real-time no micro-batching

Ohhh… that’s tricky. Have you looked at Confluent? They have some sources/sinks as well and since they are basically kafka-as-a-service they might have a better shot than most at doing realtime.

Right so it's not necessary to use any of the data connectors to do the actual data processing, what I need is the UI and Connector API libraries to work smoothly without the bloat.