hey everyone,
looking to change our ETL flow from celery alone to prefect, and i came across this discussion on github:
https://github.com/PrefectHQ/prefect/issues/1689
we are looking to have the easiest transformation into prefect and would like to have minimum changes, but i could not find a way to use a CeleryExecutor like Dask has.
Am i missing something? is it supported or should i implement it on my own?
Thank you 🙂
a
Anna Geller
03/21/2022, 11:20 AM
Without making any promises here, I believe you can achieve somewhat similar functionality using Prefect 2.0 with work queues.
We don't directly have support for Celery, but as mentioned in this linked issue, if you would want to build that, we would help to review and merge such a new execution backend. But given that Prefect 2.0 is the "LTS product", it would be interesting to see how you could implement it with Orion and together with work queues.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.