https://prefect.io logo
Title
b

Braun Reyes

05/10/2023, 7:53 PM
hi ya! does anyone use Prefect for a background worker type of pattern? As like a Celery or queue/serverless function alternative?
Worker Pool taking the place of SQS for example
j

Jai P

05/10/2023, 8:53 PM
👋 my team does this, and it has worked pretty well for us so far
we haven't amped it up to full scale yet, but generally we have an internal service that sits in front of the prefect api, takes requests from other systems, and then runs deployments (passing in something like a request id) using the prefect api. the prefect agent then is responsible for spinning up a pod in an eks cluster, and that pod has the flow running in it (which communicates with the starting service for any type of information).
b

Braun Reyes

05/10/2023, 8:56 PM
very cool
thank you for sharing
j

Jai P

05/10/2023, 9:00 PM
np! i'm sure someone from prefect knows about other cases than ours 🙂. the biggest things to consider from my point of view when you compare to celery are all the canvas methods and how they map to flows/tasks (e.g. if i had a
chord
before, how do i represent the
chord
now...is it a
@task
that needs to be run with a
dask
cluster to get the same type of parallelization, is it a
@flow
with its own deployment am i fine with it just being a concurrent process, etc. etc.)