Hey everyone. I’m wondering if anybody here has integrated Singer.io’s taps and targets with a Prefect Flow?

I haven’t myself, but since the API is generic it might be something we could write a task library abstraction over. Am I understanding right that it would be convenient for you to use singer’s taps, but you need your taps to be dependent on each other and so you want to use Prefect’s flow API? It looks like their hosted product Stitch doesn’t really do the DAG thing if I’m reading this correctly

thanks for replying! Yeah, taps are essentially data sources, and targets are sinks. By default, they’re executed as command line scripts, and they’re connected through pipes.

Probably the biggest caveat is that Prefect pipes data through the flow in memory against an in memory cache. It is acceptable, but bound by resource constraints. Interestingly, we are just starting work on extending this so that result data can be read off disk lazily when needed. I agree chained ShellTasks could work, with dataflow especially if you or your tap writes your data to stdout (see the caveat in the “Return” section of https://docs.prefect.io/api/latest/tasks/shell.html#shelltask). I agree on the dataflow point, that was my big beef too 🙂

Awesome, thanks a lot!