Apologies in advance for quite a vague question 🙂 I've historically custom rolled data pipelines (janky shell/python scripts, slightly nicer but terrible observability with Camel).
Is there much usage, or nice ways for dealing with time series and spatial data in Prefect? I can see for example being able to look at success/fail rates of the run history, I'm wondering how this could work when sometimes you only "care" about a failure for a short amount of time and that's subject to when it last failed.
Use case: satellite images are processed for various areas - there could be one job for each tile of interest. Periodically each tile is checked for new data and when there is some, a number of steps to load and process are performed.
Sometimes there isn't data, so the first step fails. This can be because it isn't available yet, or it won't ever be available. Failed might be fine because there's nothing we can do anyway (there's just no data). However if the same one is failing for say a couple of weeks and there's no "recent" data then it becomes bad.
Sometimes bad things happen and everything fails. Maybe an API key has been revoked or a service is down. Failed is bad.
For my uses I ended up tracking these and making a dashboard showing e.g. each day along an x axis and each tile along a y axis to be able to spot gaps which hopefully might go from red to green after retries.
Is something like that possible via Prefect? To pull out and create custom dashboards like that or via API calls etc?
This is purely an "out of interest" rather than an immediate need, observability and even being able to categorise failures was a huge problem so I'm just wondering if this is something catered for, so if there are solutions that would be great to here, but if not "nope, you're still on your own" is fine 😄