Just wanna share the existence of this (NEW) open source python library for loading data (from json to sql) with automatic normalisation and schema evolution how schema evolution works: https://dlthub.com/docs/reference/explainers/schema-evolution existing pipelines https://dlthub.com/docs/dlt-ecosystem/verified-sources pipeline building guide that gives you an idea of what is possible https://dlthub.com/docs/getting-started/build-a-data-pipeline comes with versioning too It is very scalable, so it can run on tiny cloud functions or large machines and utilize resources well It can of course run on prefect 🙂 Suports these destinations, https://dlthub.com/docs/dlt-ecosystem/destinations next release adds snowflake and parquet to storage destinations feedback welcome the paradigm it is for: https://dlthub.com/docs/blog/automating-data-engineers