Maryam Veisi
02/27/2023, 7:29 AMAustin Weisgrau
02/27/2023, 4:34 PMMaryam Veisi
02/27/2023, 4:43 PMAustin Weisgrau
02/27/2023, 7:04 PM@flow
def run_pipeline(parameters):
extracted_data = flow_1(parameters)
prepared_data = flow_2(extracted_data)
training_set = flow_3(prepared_data)
to execute this flow one time, you'd run it like normal python code
import run_pipeline
my_parameters = {'something': something}
run_pipeline(my_parameters)
Maryam Veisi
02/27/2023, 7:46 PMAustin Weisgrau
02/27/2023, 7:47 PMMaryam Veisi
03/01/2023, 4:06 PMAustin Weisgrau
03/01/2023, 4:31 PM#!/bin/bash
python /path/to/flow.py &
python /path/to/flow.py &
python /path/to/flow.py &
...
If you want to trigger the flows using the API and have prefect agents carry out the execution, then you need prefect agents running somewhere. You could have a bunch of prefect agents running on the same machine, or set up cloud infrastructure with one or more prefect agents running on each virtual machine.
To run 1000 flows at the same time, you would likely not be able to do that on the same machine, and need to set up cloud infrastructure to handle the load