I have the following pub-sub use case to solve with Prefect flows :
• Task A triggers a service via HTTP. Upon successful invocation, the service returns a MQ topic which needs to be subscribed to.
• Task B needs to subscribe to this topic (via MQ). The service (it is a long running process) asynchronously publishes on this topic which the Prefect task waits for and then perform subsequent tasks.
What are the best ways to do Task B in Prefect?
1 year ago
I feel there is no single answer to this, as it depends very much on details. For instance, how long does this asynchronous task usually takes? If it takes a couple of minutes, You can just wait within an upstream task, listening to MQ. But if it takes long, you are wasting time and money making your flow sit idle waiting for an async event.
I personally like to keep listening to async events external to prefect. A listener starts a prefect flow through prefect server when it receives an event from MQ.
In short, im biased to keep prefect flows as synchronized code, and make flows communicate in async fashion, not tasks.