Hi all, I am a new user:
I want to handle the use case of watching a directory for changes in zipfile metadata. For changed zipfiles several tasks would be run then. The resulting flow would be run every 24 hours.
The coarse concept of a flow could be:
1) Initialize the prefect-cache with zipfile metadata (when the flow is started)
2) At midnight get up to date zipfile metadata and compare with cached metadata
3) Refresh cache for changed zipfile metadata
4) for changed zipfile metadata only download the zipfiles and compute various derivatives
5) wait for next midnight
The part "download the zipfile and compute various derivatives" is working nicely already.
I would like to obtain recommendations referring designing a cache validator, and how to initialize the complete cache, after the flow is started.