Hi guys, we have a general questions. We are considering Prefect 2.0 (not cloud) but question about the Queue - what's the queue engine in the backend? more specifically, if we need to transfer 1000 files (or 10000) - do we need to add external queue like rabbit or kafka or can prefect handle it good? would love your 2 cents folks! thanks

You can think of a work queue as a filter. There is no RabbitMQ or anything similar - the queue is managed via backend API and SQLite/Postgres DB to simplify the architecture and limit dependencies (and limit things that can go wrong) check this for more details

Thanks it's someone that did it not an app 🙂

nice!

that was my understanding that Postgres is instead of queue... I guess I need to start some testing and see how it works. but good to know you believe it can handle thousands of files without loosing anything

correct - with self-hosting, we can't provide any guarantees, but I'm positive that running Orion on a VM with enough memory, CPU, and storage + connecting to a powerful enough Postgres should work fine here