Taylor Curran
06/22/2023, 12:20 PM@flow
def partition_ingestion_unique_pods(dataset_size, start_date, end_date, n_partitions):
date_delta = end_date - start_date
sub_flow()
partition_period = date_delta / n_partitions
for i in range(n_partitions):
end_date = start_date + partition_period
start_date = start_date + partition_period
run_deployment(
deployment="ingest_raw_orders/k8s_deployment",
parameters={
"start_date": start_date,
"end_date": end_date,
"dataset_size": dataset_size,
},
)
Interpreting results from run_deployments:
c = run_deployment("child-flow-c/dep-child-c")
value = c.state.result()
https://github.com/taylor-curran/flow-patterns/blob/main/flows/subflows/task_wrapped_deployments.py#L46Taylor Curran
06/22/2023, 12:22 PM