Hi guys we have a general questions We are considering Prefe

Question

Hi guys we have a general questions We are considering Prefect 2 0 not cloud but question about the Queue what s the queue engine in the backend more specifically if we need to transfer 1000 files or 10000 do we need to add external queue like rabbit or kafka or can prefect handle it good would love your 2 cents folks thanks

Anna Geller · Answer

You can think of a work queue as a filter There is no RabbitMQ or anything similar the queue is managed via backend API and SQLite Postgres DB to simplify the architecture and limit dependencies and limit things that can go wrong check <https discourse prefect io t what are the components of prefect 2 0 architecture 909|this> for more details

Anna Geller · Answer

given the asynchronous engine powering Prefect 2 0 API Server I believe your use case should work but you would need to use some strong Postgres DB

Anna Geller · Answer

Nice profile image btw what did you use to create it I m curious looks nice

Anat Tal Gagnon · Answer

Thanks it s someone that did it not an app slightly smiling face

Anna Geller · Answer

nice

Anat Tal Gagnon · Answer

that was my understanding that Postgres is instead of queue I guess I need to start some testing and see how it works but good to know you believe it can handle thousands of files without loosing anything

Anna Geller · Answer

correct with self hosting we can t provide any guarantees but I m positive that running Orion on a VM with enough memory CPU and storage + connecting to a powerful enough Postgres should work fine here