@Jeremiah could you write more about what is in store with this?
My current use-case is quite simple and I think that Prefect covers it currently.
I have an input, which is either a query or file with some datapoints (there could be from 1 to xxx million datapoints). So here I need conditional tasks, based on input. After this I need to split those datapoints to 10k batches and do some procesing on each batch. So there can be 1 to xx thousand batches and here I need to create a task for each batch.
I will be working on this in the next couple of weeks and I hope everything works out okay, as I already promised presentation about our Airflow and Prefect journey to local Python meetup 🙂
Currently, after cursory research, I see couple things that is harder for me currently than Airflow.
1. Kubernetes deployment. I understand Helm charts and operators, but I have not found anything in that regard with Prefect, so Kubernetes deployment is not clear for me (could be that I didn’t do enough research yet)
2. Secret storage. I know that Prefect supports secret storage if using your cloud solution, but we are looking into full on-premises deployment currently.
3. UI authentication. This is similar to 2 point.
4. Prefect seems a bit less mature than Airflow, but it is not a deal breaker for me.