Slackbot
05/21/2024, 8:08 PMMarvin
05/21/2024, 8:09 PMconcurrency
or Flow.submit
to control parallel execution.
### Execution Order in Prefect
- Tasks in a flow execute in the order they are submitted unless specifically controlled asynchronously or through concurrency management.
- If you call StifelLoader()
from within StifelFileRetriever()
, StifelLoader
will be sequentially executed after the preceding code in StifelFileRetriever
.
- Tasks or subflows can be executed in parallel by utilizing concurrent task runners
or submitting flows from within another flow to allow for concurrent execution.
Relevant Documentation and Discussion:
- Flow run example of tasks executed sequentially:
Flow run dependency order
- Managing task concurrency within a flow:
Concurrency handling
- Submit subflows to run in the background (potential enhancement):
GitHub feature discussion
### Your Code
Given your code:
python
@flow(name='loaders-sra-stifel')
def StifelLoader():
logger = loggers.get_run_logger()
try:
inceptionUrl = GetInceptionServiceUrl()
<http://logger.info|logger.info>(f'Inception Service Url: {inceptionUrl}')
post_request_process(inceptionUrl, 'StifelLoader', None, 300)
if time_is_later_than_7AM():
SRALoaderAlerts()
except Exception as err:
logger.error(f'Error from Stifel Loader: {err}')
@flow(name='file-retriever-stifel')
def StifelFileRetriever():
logger = loggers.get_run_logger()
try:
inceptionUrl = GetInceptionServiceUrl()
<http://logger.info|logger.info>(f'Inception Service Url: {inceptionUrl}')
post_request_process(inceptionUrl, 'StifelFileRetriever', None, 300)
StifelLoader() # This line ensures sequential execution of StifelLoader after previous steps in StifelFileRetriever
except Exception as err:
logger.error(f'Error from Stifel File Retriever: {err}')
StifelLoader
is called sequentially after the previous steps in StifelFileRetriever
. If you want parallel execution, you would need to use concurrent execution tools available in Prefect.
Would you like to see an example of converting these flows to run in parallel?