Slackbot
05/21/2024, 8:08 PMMarvin
05/21/2024, 8:09 PMconcurrency
or Flow.submit
to control parallel execution.
### Execution Order in Prefect
- Tasks in a flow execute in the order they are submitted unless specifically controlled asynchronously or through concurrency management.
- If you call StifelLoader()
from within StifelFileRetriever()
, StifelLoader
will be sequentially executed after the preceding code in StifelFileRetriever
.
- Tasks or subflows can be executed in parallel by utilizing concurrent task runners
or submitting flows from within another flow to allow for concurrent execution.
Relevant Documentation and Discussion:
- Flow run example of tasks executed sequentially:
Flow run dependency order
- Managing task concurrency within a flow:
Concurrency handling
- Submit subflows to run in the background (potential enhancement):
GitHub feature discussion
### Your Code
Given your code:
python
@flow(name='loaders-sra-stifel')
def StifelLoader():
logger = loggers.get_run_logger()
try:
inceptionUrl = GetInceptionServiceUrl()
<http://logger.info|logger.info>(f'Inception Service Url: {inceptionUrl}')
post_request_process(inceptionUrl, 'StifelLoader', None, 300)
if time_is_later_than_7AM():
SRALoaderAlerts()
except Exception as err:
logger.error(f'Error from Stifel Loader: {err}')
@flow(name='file-retriever-stifel')
def StifelFileRetriever():
logger = loggers.get_run_logger()
try:
inceptionUrl = GetInceptionServiceUrl()
<http://logger.info|logger.info>(f'Inception Service Url: {inceptionUrl}')
post_request_process(inceptionUrl, 'StifelFileRetriever', None, 300)
StifelLoader() # This line ensures sequential execution of StifelLoader after previous steps in StifelFileRetriever
except Exception as err:
logger.error(f'Error from Stifel File Retriever: {err}')
StifelLoader
is called sequentially after the previous steps in StifelFileRetriever
. If you want parallel execution, you would need to use concurrent execution tools available in Prefect.
Would you like to see an example of converting these flows to run in parallel?Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by