Hello everyone,
I wonder if Prefect allows me to do this feature I will describe.
I am currently working on a project requiring a "central" notification system to send overall progress and aggregate information about the flow and sub-tasks to an external service.
For example, my main flow dispatches a single task, such as "Acme", multiple times with different inputs (Linux, Windows, Mac, etc.).
These tasks trigger other tasks in batches, such as processing a list of 200 items in batches of 20 for Linux.
I need to send http call to an external service with the overall progress of the whole flow, such as the percentage of completion for each platform.
Flow overall: 10%
Linux: Processing 20%
Macos: Processing 50%
...
I question whether Prefect can provide an out-of-the-box solution for this notification system or if anyone could recommend any resources or best practices to help me achieve this.
I know that Prefect has callbacks for each task when it concludes, but I am wondering if there is a way for my main flow to track the state changes of the sub-tasks as well.
Thank you in advance for any guidance or suggestions you may have.