https://prefect.io logo
#best-practices
Title
# best-practices
a

Anat Tal Gagnon

06/08/2022, 12:59 PM
Hi guys, we have a general questions. We are considering Prefect 2.0 (not cloud) but question about the Queue - what's the queue engine in the backend? more specifically, if we need to transfer 1000 files (or 10000) - do we need to add external queue like rabbit or kafka or can prefect handle it good? would love your 2 cents folks! thanks
1
a

Anna Geller

06/08/2022, 1:04 PM
You can think of a work queue as a filter. There is no RabbitMQ or anything similar - the queue is managed via backend API and SQLite/Postgres DB to simplify the architecture and limit dependencies (and limit things that can go wrong) check this for more details
given the asynchronous engine powering Prefect 2.0 API Server, I believe your use case should work, but you would need to use some strong Postgres DB
Nice profile image btw - what did you use to create it? I'm curious - looks nice
a

Anat Tal Gagnon

06/08/2022, 1:07 PM
Thanks it's someone that did it not an app 🙂
a

Anna Geller

06/08/2022, 1:08 PM
nice!
a

Anat Tal Gagnon

06/08/2022, 1:08 PM
that was my understanding that Postgres is instead of queue... I guess I need to start some testing and see how it works. but good to know you believe it can handle thousands of files without loosing anything
a

Anna Geller

06/08/2022, 1:11 PM
correct - with self-hosting, we can't provide any guarantees, but I'm positive that running Orion on a VM with enough memory, CPU, and storage + connecting to a powerful enough Postgres should work fine here
🙌 1
5 Views