hi ya! does anyone use Prefect for a background worker type of pattern? As like a Celery or queue/...

Braun Reyes

05/10/2023, 7:53 PM

hi ya! does anyone use Prefect for a background worker type of pattern? As like a Celery or queue/serverless function alternative?

Braun Reyes

05/10/2023, 7:54 PM

Worker Pool taking the place of SQS for example

Jai P

05/10/2023, 8:53 PM

👋 my team does this, and it has worked pretty well for us so far

Jai P

05/10/2023, 8:55 PM

we haven't amped it up to full scale yet, but generally we have an internal service that sits in front of the prefect api, takes requests from other systems, and then runs deployments (passing in something like a request id) using the prefect api. the prefect agent then is responsible for spinning up a pod in an eks cluster, and that pod has the flow running in it (which communicates with the starting service for any type of information).

Braun Reyes

05/10/2023, 8:56 PM

very cool

Braun Reyes

05/10/2023, 8:57 PM

thank you for sharing

Jai P

05/10/2023, 9:00 PM

np! i'm sure someone from prefect knows about other cases than ours 🙂. the biggest things to consider from my point of view when you compare to celery are all the canvas methods and how they map to flows/tasks (e.g. if i had a

chord

before, how do i represent the

chord

now...is it a

@task

that needs to be run with a

dask

cluster to get the same type of parallelization, is it a

@flow

with its own deployment am i fine with it just being a concurrent process, etc. etc.)

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.