Hi, does anyone have recommendations or best practices that they follow around dependencies? My specific example has to do with extraction/load vs transformation steps. It is convenient to put it all in one Flow so that the transformations start after the extraction/loads are complete. But our Flows have become quite monolithic, and we want to be able to test various sections (i.e. extract/load) separately from other steps (i.e. transformations). Any recommendations here?
a
Anna Geller
12/15/2021, 4:52 PM
@Adam Roderick you could explore the flow of flows pattern. This allows you to run the extract-load jobs separately from transformations when needed, while at the same time allowing to call all child flows in a parent flow that can run on schedule. We wrote about this in this blog post.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.