Hello everyone. I have multiple microservices/projects (look at attached image) for use cases like Q&A generation, Search, Data Ingestion, etc. If we were to provide devs the ability to combine these individual components to stitch together as a service, would Prefect be of help here?
07/12/2022, 9:45 AM
Hi! It depends to a large extent on what each of these projects is doing - if those apps are e.g. APIs from which you are serving some models for predictions, then you'd need some other tool for that. But for use cases such as data ingestion that you mentioned, Prefect can help orchestrate those for sure
Laxman Singh Tomar
07/12/2022, 11:43 AM
thanks @Anna Geller, so basically within each of these microservices/projects, Prefect can be used for running a pipeline. But, it cannot be used if I want to stitch together x functionality from one project, with y functionality of another project.
07/12/2022, 11:55 AM
you can totally stitch together the functionality of multiple projects and Prefect is actually a perfect fit for that given that Prefect 2.0 is the only open-source orchestrator I'm aware of which natively supports subflows https://orion-docs.prefect.io/concepts/flows/#subflows
You can have a parent flow that triggers one child flow for each of those projects if they need to be stitched together