William Jamir

05/02/2023, 1:34 PM
Hello everyone, I wonder if Prefect allows me to do this feature I will describe. I am currently working on a project requiring a "central" notification system to send overall progress and aggregate information about the flow and sub-tasks to an external service. For example, my main flow dispatches a single task, such as "Acme", multiple times with different inputs (Linux, Windows, Mac, etc.). These tasks trigger other tasks in batches, such as processing a list of 200 items in batches of 20 for Linux. I need to send http call to an external service with the overall progress of the whole flow, such as the percentage of completion for each platform.
Flow overall: 10%
Linux: Processing 20%
Macos: Processing 50%
I question whether Prefect can provide an out-of-the-box solution for this notification system or if anyone could recommend any resources or best practices to help me achieve this. I know that Prefect has callbacks for each task when it concludes, but I am wondering if there is a way for my main flow to track the state changes of the sub-tasks as well. Thank you in advance for any guidance or suggestions you may have.