Kevin Jacobs
05/13/2022, 11:22 AMAnna Geller
@flow(name="Streaming Pipeline with Tasks")
def main():
orders = get_real_time_data()
load_to_dwh(orders)
if __name__ == "__main__":
while True:
main()
time.sleep(5)
The benefit of this approach is that you get way more visibility and observability, which can be especially valuable when things go wrong.Kevin Jacobs
05/13/2022, 12:10 PMAnna Geller
time.sleep()
could be milliseconds if needed. The main idea is to get more visibility into which streams succeeded and which didn't