Silly question, where does Prefect fit in with Python Celery and Message Queues / Brokers (e.g. RabbitMQ)?
It seems like some of the tasks, such as restarting (or retrying) a long running asynchronous job can be implemented in either of the above solutions. I was wondering what would be the difference and the tradeoffs to using each of the above solutions.
s
Shawn Holyoak
01/24/2023, 5:50 PM
I don't think it's a silly question. I'm evaluating Prefect as a Celery replacement. It seems so far that Prefect does what Celery does, but it has better observability, better state tracking, easier complex job creation (multiple tasks chained in Celery vs Flows in Prefect), and very nice UI (which requires (celery-flower for Celery). Seems like a really nice upgrade to Celery so far to me.
o
Ofir
01/24/2023, 5:51 PM
Thanks @Shawn Holyoak I appreciate the honest response!
👍 1
Under the hood, both Prefect and Celery have persistency in some shape or form (e.g. PostgreSQL) and message queues in their underlying implementation?
z
Zanie
01/24/2023, 5:52 PM
We’re actively investigating ways to be a better drop-in replacement for Celery 😉 I bet our product team would be interested in chatting with ya’ll about your needs.
:upvote: 2
o
Ofir
01/24/2023, 5:52 PM
Thanks @Zanie, I’d love that 🙂
s
Shawn Holyoak
01/24/2023, 5:53 PM
Plus 1 for me
j
justabill
01/24/2023, 5:54 PM
@Ofir and @Shawn Holyoak please DM me your email address and I'll find some time for us to chat. As @Zanie said, we're actively exploring how Prefect could better solve the same problems that people currently use Celery for.