moti shakuri

03/21/2022, 11:04 AM
hey everyone, looking to change our ETL flow from celery alone to prefect, and i came across this discussion on github: we are looking to have the easiest transformation into prefect and would like to have minimum changes, but i could not find a way to use a CeleryExecutor like Dask has. Am i missing something? is it supported or should i implement it on my own? Thank you 🙂

Anna Geller

03/21/2022, 11:20 AM
Without making any promises here, I believe you can achieve somewhat similar functionality using Prefect 2.0 with work queues. We don't directly have support for Celery, but as mentioned in this linked issue, if you would want to build that, we would help to review and merge such a new execution backend. But given that Prefect 2.0 is the "LTS product", it would be interesting to see how you could implement it with Orion and together with work queues.

moti shakuri

03/21/2022, 3:02 PM
thank you anna 🙂
👍 1