https://prefect.io logo
t

Thierry

07/25/2023, 8:47 PM
Hi everyone! My name is Thierry and I'm a developer advocate for the open-source project Hamilton. While Prefect is an orchestration tool that helps run code on infra (macro), Hamilton is a micro-framework that help you write data transformations in a principled manner (think dbt for Python). A core challenge of building a workflow is determining how to break it down into tasks. As your project grows, complex you have to find the right trade-off between the complexity of your workflow structure and the complexity of each task. At the orchestration level, adding too many tasks can add overhead and be operationally costly. In contrast, Hamilton encourages very granular data transforms and automatically creates the execution DAG for you! I just published a blog post on how Prefect and Hamilton complement each other! By adding Hamilton to your Prefect task, you get easier to test and reuse data transformations. Also, it can nicely integrate with Prefect's Block and Artifact to improve the expressivity of your workflows and your data lineage. Happy to hear your thoughts and chat about this problem space!
gritty intensifies 6
👏 5
🙌 8
👀 6