I am running into multiple use cases where it seems like a pub/sub pattern could be useful. Within a flow, I know that I can start multiple subflows, but that requires me to know beforehand which subflows might pertain to the main flow's output result. Is there a pub/sub like pattern in prefect that would allow certain flows to "subscribe" or be triggered by the completion of other flows without manually specifying the relationship in the original flow?

Hello @Richard Alexander, You can use different breakdowns of tasks and flows or subflows and flows. A flow is the lowest level you are able to have easy visibility and interaction with the UI. To have flows trigger other flows, you can use its state to conditionally react. Here is an example: https://discourse.prefect.io/t/how-to-build-a-conditional-flow-of-flows-i-e-trigger-a-different-child-flow-depending-on-the-upstream-flows-state/202

Thanks for the reply @Sahil Rangwala! I should have mentioned that I am using prefect 2.3 (though I've been using prefect since 0.11). The example you mentioned is in prefect v1 and it is exactly the pattern that I'm trying to avoid. To be more specific, I have a very modular use case where a "controller" flow needs to kick off many "child" flows and eventually aggregate results in some way. The issue is we are building the modular "child" flows at a rapid pace and it is becoming burdensome to always go back and edit the "controller" flow when a new "child" flow is built. I completely understand how to create subflows, but that is not what I'm after. Is there some kind of pub/sub like pattern in prefect that would allow a "controller" flow to publish a request to "child" flows without knowing how many "child" flows might pick up the request? Then the "controller" would listen for results from all of the "child" flows until all of them complete and aggregate the data into a final result?

Thanks for the additional information @Richard Alexander! You would need to ensure the child flows are properly tagged initially and then can check for a certain tag for flows, for example:

child

. The controller flow would hit the api and look for the child flow with that tag '`child`' .

Ah... I see @Sahil Rangwala. What is the roadmap for tags in Prefect 2.0? It seems like tags are much less widely used versus 1.0. So I just want to make sure that tags will be sticking around if I put a lot of work into building flows with tags. 😉

Unsure of the future roadmap on tags, but we have no plan on dropping support for tags in 2.0! Hopefully this was helpful 😄

Yes, definitely. Thanks so much! 😄

Hello @Sahil Rangwala, thanks for your reply to @Richard Alexander. We got similar use cases which need pub/sub pattern support (mostly for handling stock market data). I know the solution you mention (API with tag) works, but I guess there will be a better solution to to deal with it. Wonder if there is any plan to better support pub/sub pattern?