Probably the biggest caveat is that Prefect pipes data through the flow in memory against an in memory cache. It is acceptable, but bound by resource constraints. Interestingly, we are just starting work on extending this so that result data can be read off disk lazily when needed.
I agree chained ShellTasks could work, with dataflow especially if you or your tap writes your data to stdout (see the caveat in the “Return” section of
https://docs.prefect.io/api/latest/tasks/shell.html#shelltask).
I agree on the dataflow point, that was my big beef too 🙂