Hi guys 🙂 I need connector/extractor tasks (to load data into a data lake) for several API services, starting with GoogleAds task group for interacting with the Google Ads API (connecting, pulling data). Thinking also about integrating Hubspot, Microsoft Ads, Facebook Ads, DV360, Supermetrics, and other marketing data sources. I'm writing it here so maybe I can get some guidance or if there's already some effort which I could join?
04/13/2021, 1:41 PM
Hi @Michal Zawadzki! Most of the Tasks in the Task Library are community contributed. You can check the source code for one of them. They aren’t hard to write. It’s very easy to extend Prefect. You’re also welcome to make a pull request for the Tasks you write
04/13/2021, 4:17 PM
Thanks Kevin :) Was just wondering if there are any plans for the task library other than just adding new tasks. I'll be happy to contribute anyhow.
I was also wondering if there is some vision behind this library that we would fit in with these connectors. Eg. there are many data extraction services that basically take data from different sources and put it into a data lake. This could be done much much cheaper with eg. Prefect.
04/13/2021, 5:29 PM
As long as you can do it in Python, it can be done in Prefect. Yes it can be cheaper, although to be fair, those services manage the compute needed to move the data while we don’t.
Yes the goal is just to get a lot of Tasks so we can have more “batteries included” for out users. Task contributions are definitely welcome 🙂
04/13/2021, 6:58 PM
They do it but they add a significant margin over what you can accomplish even using a reserved VM setup. I think this stuff can be done an order of magnitude cheaper. You can already see it with some proprietary tools like Azure's Copy Tool. I think it can eventually be done with Prefect as well.