I think I'm trying to wrap my head around whether Prefect is a tool that is good for our very specific problem. When I read about the case studies, etc. it sounds like it is solving different problems.
An example of our problem and what I'm trying to solve:
We have a multi-terabyte drive of data we need to upload to the cloud.
We upload it to the cloud. We run multiple proprietary tools on servers to extract the data. Transform and process it multiple times. Say 20 different steps in one workflow.
The process will break multiple times during this process and we need to inspect and potentially even manually fix problems inside the files themselves and debug / rerun individual steps.
So none of this is ever customer data, we're never trying to gain insight about big data generated from users or anything. This is purely a technical GIS data processing project.