hey all, i have a question about infrastructure for prefect 2.
my use case is that in general, we run a small number of flows daily that process relatively low volumes of data. they take minutes to less than an hour to complete. occasionally, but not very often, our data volume spikes, and it takes hours to complete.
right now we are on prefect 1, deployed on ECS. we have one agent as an ECS task that we've provisioned low compute. it is slow; we've throttled its amount of concurrent flow executions to avoid it freezing; but it works.
we need the ability to run our pipelines faster, and the imminent migration to prefect 2 is the right moment to figure this out. the docs
recommend we jack up the ECS task's compute. but 1) we don't know our volume in advance so a spike could require more than we expected, and 2) it doesn't make sense to provision a crazy amount of compute when we only need it infrequently.
i have the aws lambda model in my head, where scaling horizontally is a piece of cake. does either the docker or k8s infra implementation support this type of thing, e.g. supports N containers concurrently; spins up and kills a container for each flow run? or perhaps some combo of
ecs with fargate?
what's the best approach to horizontal scaling with prefect 2? or is this not the right approach at all?
worth mentioning here is that we use two different flow patterns:
1. one flow that does all processing << takes a long time
2. subflow pattern, where parent flows delegate chunks to work to child flows << seems to be quicker, but we quickly ran into concurrency issues with multiple flows running side by side
the plan is to migrate what we can to approach #2.