Hi,
Thanks for all the help over the past few weeks/months.
I've been evaluating prefect for a use case and right now it looks like it doesn't add much for what we do.
I wanted to check with you guys, to see if I'm missing something or if I'm on the right track.
I have a python model that is very disk intensive. We generate and move around lots of intermediate data, about 180GB, which is going to increase too.
I currently run the model using LocalDaskExecutor on a single AWS EC2 instance with about 92 cores and 196GB RAM.
While prefect gives me the ability to scale horizontally, if I were to spread the work horizontally, moving the intermediary data between instances is going to be slower than referencing off disk.
So in summary, while I can just scale vertically our model seems limited by disk access so using something like prefect to go horizontal appears to add additional overhead.
Thoughts, questions, opinions all welcome.