This will probably sounds like a weird question, but how is Prefect different from Celery?. I think the differences from Airflow are well described in the docs, yet quite clear, but I'm not sure why I should prefer it over Celery.

Hi @stefano, it’s a good question - many technologies can be used together, and in fact it would be straightforward to run Prefect on top of a Celery backend (though we prefer Dask for the same purpose). Celery is a system for executing work, usually in a distributed fashion. A “broker” submits work to a pool of workers, who run the task/job/function and indicate that they’ve finished. So Celery’s primary job is to take a function and run it. Dask isn’t exactly the same but similar enough for the purpose of this conversation. Prefect, in contrast, is a workflow management system. Prefect’s primary job is to supply workflow semantics around your functions — things like what to do if they fail, how to handle logging, retries, conditional logic, responding to upstream results, complex dependencies, scheduling, etc. If all of that workflow logic “passes”, Prefect then runs your task on whatever backend you prefer, which is where Celery could come in to play. Currently, Prefect ships with local, multiprocessing, threaded, and Dask distributed backends, and there’s an outstanding PR for a redis-queue backend. Celery would fit nicely as another execution backend.

thank you!

Thank you @Jeremiah, very helpful for newbie like me who's overwhelmed by the choices/categorizations and don't know what's different between each

@Chris White we should archive this thread with Marvin

@Marvin archive “What is the difference between Prefect and Celery?”