Hello everyone,
I'm trying to parallelize flows execution:
Copy code
@flow(log_prints=True, task_runner=ConcurrentTaskRunner)
def sample_flow(sample_index) -> None:
...
def samples_parallel_executer(sample_indices):
for sample_index in sample_indices: #TODO: Add parallelism
sample_flow(sample_index)
However, it wasn't obvious for me how to parallelize flows execution in Prefect. Any idea what are the best practices?
Thanks in advance for your help!
i
Igor
09/30/2024, 9:02 AM
Hey, I think that you can make your flows async and call asyncio.gather, or do sample_flow(return_state=True) and check the state of these flows
m
Maroun Khriesh
10/01/2024, 6:21 AM
Can you clarify what do you mean by "do sample_flow(return_state=True) and check the state of these flows"?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.