Hi, we’ve got a flow of flows, which runs every time a file is added and then at the end runs a flow that updates a dataset in Power BI. Since we’re only able to refresh PowerBI 8 times a day, is there a way to group the power bi refreshes together in case multiple files are added? Just wondering what might be the best way to architect that, or if anyone’s done something similar
05/25/2022, 12:55 PM
iterating some business logic over a runtime-discoverable list of files looks like a perfect use case for mapping - did you try that?
alternatively, you could have a flow that fires any time a new file is added in an event-driven way
if I misunderstood your question, could you explain in more detail what you are trying to do?
to group multiple flow runs or task runs, in Prefect 2.0 there is a flexible concept of tags and the UI provides filter functionality allowing you to even build custom dashboards based on that - this topic may help
05/25/2022, 12:59 PM
Thanks! We have it set up in an event driven way so that a new flow is triggered every time a file is added. The problem is when a new file is added and then 2 mins later another one, and so on…
Ideally it would be something like merging flows, or have the powerbi refresh on some kind of a delay
Then if the refresh flow had already been triggered and was pending it would not allow it to trigger again
05/25/2022, 2:35 PM
I think there are things you can do there, but the workload sounds too unpredictable. Is this Prefect 1 or 2? For Prefect 1, you can use the state handler to check for the last flow run and if it’s too recent, cancel the current run.
The “merging” is slightly possible with that state handler. The parent flow can also schedule the child flow with a delay. But the problem is you don’t know the exact delay here to use. And I think going this route makes the architecture far more complicated that trying to bump up the 8 refreshes a day would be significantly less work.