Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
w
wiretrack
04/08/2022, 9:53 PMHi all. Wondering if anybody have experienced running Prefect inside an API context using a third party worker, like Celery or something. What I mean is, declaring the flow/tasks with Prefect Core, and then calling
flow.run()k
Kevin Kho
04/08/2022, 9:55 PMI thought Prefect or Celery was an either-or thing. Havent come across them being used together but I guess you could yep if it works. We’ll see if anyone in the community chimes in
w
wiretrack
04/08/2022, 9:57 PMYeah, we actually have been using Prefect with the whole server running in Kubernetes, but that's more like an ETL structure, and that works great.
For a new project, I was trying to run away from Celery chains and chords, so it could make sense to declare the flow using Orion. But then I would need something to actually run the flow without the server overhead.
But that's not something I've heard or seen anybody doing yet, so not too sure if it's a good idea
we haven't explored orion much, but on 1.0 the server was fairly complex, and not really suited for this kind of approach, way too many pieces to tie together.
if Orion turns out to be more flexible, we could be a awesome to use it with a custom executor (Celery, RQ, Huey etc). Monitoring would be great and definitely a game changer when comparing to current tools (thinking of Flower and RQ Dashboard).
Having the ability to write proper flows (one flow can be an executor task, or maybe each flow task can be an executor task) is also great, and not many options for that at the moment.
Understand this is not exactly what it was designed to do, but could be an option
k
Kevin Kho
04/09/2022, 4:34 AMYeah I think orchestrating with a backend will just have additional overhead so I can see why you want to do this. But I guess the question is why use Celery over bringing over the work to Prefect + Dask? Is it because you were already using Celery?
w
wiretrack
04/09/2022, 4:59 AMActually it's because of our experience with 1.0. Everything works great, but its just too much. There were 7-8 services to lookout for on K8S. I can make Celery work with 1 worker and that's it (i already have RabbitMQ/Redis anyways). It definitely works great when the job itself is a whole ETL flow. But when doing background jobs (that do have the "workflow" aspect), as it is, is just too much.
k
Kevin Kho
04/09/2022, 5:04 AMAh yeah I don’t think Prefect 1.0 is meant for these batch jobs. I can see what you’re trying to do though because there are useful Flow level semantics you still get with
flow.run()