hi ya does anyone use Prefect for a background worker type o

Question

hi ya does anyone use Prefect for a background worker type of pattern As like a Celery or queue serverless function alternative

Braun Reyes · Answer

Worker Pool taking the place of SQS for example

Jai P · Answer

wave my team does this and it has worked pretty well for us so far

Jai P · Answer

we haven t amped it up to full scale yet but generally we have an internal service that sits in front of the prefect api takes requests from other systems and then runs deployments passing in something like a request id using the prefect api the prefect agent then is responsible for spinning up a pod in an eks cluster and that pod has the flow running in it which communicates with the starting service for any type of information

Braun Reyes · Answer

very cool

Braun Reyes · Answer

thank you for sharing

Jai P · Answer

np i m sure someone from prefect knows about other cases than ours slightly smiling face the biggest things to consider from my point of view when you compare to celery are all the canvas methods and how they map to flows tasks e g if i had a `chord` before how do i represent the `chord` now is it a ` task` that needs to be run with a `dask` cluster to get the same type of parallelization is it a ` flow` with its own deployment am i fine with it just being a concurrent process etc etc