Taylor Curran
@flow def partition_ingestion_unique_pods(dataset_size, start_date, end_date, n_partitions): date_delta = end_date - start_date sub_flow() partition_period = date_delta / n_partitions for i in range(n_partitions): end_date = start_date + partition_period start_date = start_date + partition_period run_deployment( deployment="ingest_raw_orders/k8s_deployment", parameters={ "start_date": start_date, "end_date": end_date, "dataset_size": dataset_size, }, )
c = run_deployment("child-flow-c/dep-child-c") value = c.state.result()
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.