Hi,
We're thinking about using prefect for writing our "Flows".
For very large workloads, we're looking for a distributed solution and we're so happy that prefect has such a smooth integration with dask. :)
However, we also have workloads that are extremely light and require sub-second latency. We performed some benchmarks internally and realized that compared to bare-metal python, prefect has an overhead of about 50ms - which is okay for us.
But, we were thinking that as prefect grows, is there a possibility that perhaps this overhead would grow too big (in seconds)?
Ultimately, this is more of a question regarding the prefect philosophy. Is prefect going to concentrate only on large workloads that require throughput while sacrificing the latency for small workloads? Or, is it going to keep a balance between both worlds?
Thanks!