https://prefect.io logo
Title
o

Ofir

01/24/2023, 5:39 PM
Silly question, where does Prefect fit in with Python Celery and Message Queues / Brokers (e.g. RabbitMQ)? It seems like some of the tasks, such as restarting (or retrying) a long running asynchronous job can be implemented in either of the above solutions. I was wondering what would be the difference and the tradeoffs to using each of the above solutions.
s

Shawn Holyoak

01/24/2023, 5:50 PM
I don't think it's a silly question. I'm evaluating Prefect as a Celery replacement. It seems so far that Prefect does what Celery does, but it has better observability, better state tracking, easier complex job creation (multiple tasks chained in Celery vs Flows in Prefect), and very nice UI (which requires (celery-flower for Celery). Seems like a really nice upgrade to Celery so far to me.
o

Ofir

01/24/2023, 5:51 PM
Thanks @Shawn Holyoak I appreciate the honest response!
👍 1
Under the hood, both Prefect and Celery have persistency in some shape or form (e.g. PostgreSQL) and message queues in their underlying implementation?
z

Zanie

01/24/2023, 5:52 PM
We’re actively investigating ways to be a better drop-in replacement for Celery 😉 I bet our product team would be interested in chatting with ya’ll about your needs.
:upvote: 2
o

Ofir

01/24/2023, 5:52 PM
Thanks @Zanie, I’d love that 🙂
s

Shawn Holyoak

01/24/2023, 5:53 PM
Plus 1 for me
j

justabill

01/24/2023, 5:54 PM
@Ofir and @Shawn Holyoak please DM me your email address and I'll find some time for us to chat. As @Zanie said, we're actively exploring how Prefect could better solve the same problems that people currently use Celery for.
🙌 4