Would Prefect (cloud) be a good choice for a "queue of async logic" type of workload? It's not a typical scheduled data ETL pipeline scenario but instead it's small parts of logic that are triggered by actions of users of your app. For example, in our app, when a user purchases a coupon we want to start a flow that performs some logic (some DB and API calls) in a durable fashion. Specifically, we would use the ECS push work pools. So let's say tens, hundreds of new ECS tasks per minute. I'm wondering if this is a viable use case of Prefect (performance and cost wise), compared to something more traditional like, let's say, a kafka topic with a consumer that performs this logic asynchronously? Thanks!

Great question @Yaron Levi! I think it’s highly likely that you would run into AWS rate limits when creating ECS tasks at that rate. At that scale, I think you’d want a fleet of instances all serving your flows with

flow.serve

or

serve,

especially if you expect a consistently high load.

Adding to what Alex said, we are actively exploring "task scheduling" use cases that this could fit into nicely; would you be game to chat either synchronously or asynchronously about your use case and dreams for how Prefect could help?

@Chris White Yes! I would love that...

Awesome let's do it! Would you mind emailing me at chris@prefect.io with some availability and we can take it from there?

@Chris White Sure. I've just an email.