Hey guys, has anyone considered using Celery as the executor for prefect tasks? ie, A part of the workflow have CPU intensive tasks and need more level of concurrent distribution, which can’t be achieved with prefect agents. So Prefect agent will submit the tasks to a message queue with necessary parameters and Celery will manage such executions. Open for Triage!

👋 ! Can you say more about your use case and why Celery came to mind? Is it more or less looking for higher task throughput or is it something in particular about celery?

It’s sort of both. We use Prefect for orchestrating data pipelines, in which the data will be acquired through scrapers. The scrapers has to be distributed on a higher level to achieve expected throughput and our legacy systems have been built considering Celery for distribution purpose.

Yep yep makes sense. I relied pretty heavily on prefect + celery in a previous role for large scale scraping, but celery was really just a stand in for “redis-backed equitable distribution across workers”. Celery, specifically, is pretty make or break for you here?

Yes, otherwise it would create a pretty huge tech debt for redesigning the scrapers. And I am pretty much intrigued to understand the implementation of prefect+Celery in your previous experience. Would you mind sharing a higher level idea?