Kevin Jacobs05/13/2022, 11:22 AM
The benefit of this approach is that you get way more visibility and observability, which can be especially valuable when things go wrong.
@flow(name="Streaming Pipeline with Tasks") def main(): orders = get_real_time_data() load_to_dwh(orders) if __name__ == "__main__": while True: main() time.sleep(5)
Kevin Jacobs05/13/2022, 12:10 PM
could be milliseconds if needed. The main idea is to get more visibility into which streams succeeded and which didn't