https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jai P

06/06/2022, 5:09 PM
👋 hullo! i'm curious, if i want to use prefect 2.0 for workflows that i'd typically do with something like celery (e.g. enabling an event-driven architecture vs. more ETL-y type workloads), is that a bad idea?
k

Kevin Kho

06/06/2022, 5:10 PM
No it’s not and it’s something we think 2.0 can handle. The only thing is that Prefect is not resource-aware so you need to explicitly set the flow concurrency yourself for you agent. Does that make sense? You can use the API to create a deployment
j

Jai P

06/06/2022, 5:13 PM
hmm, i think that makes sense! and i'd set the flow concurrency in the deployment spec?
oh, or is it
work queue concurrency
that i'd have to set
k

Kevin Kho

06/06/2022, 5:14 PM
Yes exactly
👍 1
j

Jai P

06/06/2022, 5:17 PM
awesome! is there anything else you'd suggest we look out for? to paint a picture, we have an entity that might trigger 100s of events per minute that we'd like to trigger flow runs for. generally we'd just put that behind a queue and have a celery app process the different types of events, but we have a need for things like caching/more fine-grained concurrency, and don't necessarily want to implement those features ourselves (and think using prefect could be a better solution)
k

Kevin Kho

06/06/2022, 5:21 PM
I think nothing else aside from being aware that Prefect might error if it tries to deploy a flow but doesn’t have resources
j

Jai P

06/06/2022, 5:24 PM
got it, ok! if we run into problems along the way, we'll definitely reach out. thanks for the help!
5 Views