hey folks, high level question here. I'm building a system that will hit an SQS queue for a pub/sub thing capturing point in time updates to a dataset. The flow from the queue is not terribly big, but will have a few million or more dispatches during normal working hours. I'll want to hit the queue, parse the message (the payloads are complete messages with no external info necessary, and all strings/bytes), do minor transforms, and then send an update query to a relational DB with a soft real-time limit of around one second. Is there any resources as to the utility of prefect in this kind of use case? I'm interested in using a data pipeline (vs. a boutique app) in this context because there's a lot more pipelines I need to build, and would rather circle them all up in one spot. What say ye, people of #CL09KU1K7?

Hi @William MacMillan! It seems that for this use case, you have real-time low latency queue. Prefect is intended more for batch jobs (but can be kicked off by the GraphqL API on demand). I think your SQS can feed into Lambda and Prefect doesn’t quite fit there.

Gotcha. I love the model with prefect, I'll probably still bring it in the stack instead of rolling my own or using another pipeline, but I won't consider it for this particular use case. Thanks for the insight!