👋 hullo! i'm curious, if i want to use prefect 2.0 for workflows that i'd typically do with something like celery (e.g. enabling an event-driven architecture vs. more ETL-y type workloads), is that a bad idea?
k
Kevin Kho
06/06/2022, 5:10 PM
No it’s not and it’s something we think 2.0 can handle. The only thing is that Prefect is not resource-aware so you need to explicitly set the flow concurrency yourself for you agent. Does that make sense?
You can use the API to create a deployment
j
Jai P
06/06/2022, 5:13 PM
hmm, i think that makes sense! and i'd set the flow concurrency in the deployment spec?
Jai P
06/06/2022, 5:13 PM
oh, or is it
work queue concurrency
that i'd have to set
k
Kevin Kho
06/06/2022, 5:14 PM
Yes exactly
👍 1
j
Jai P
06/06/2022, 5:17 PM
awesome! is there anything else you'd suggest we look out for? to paint a picture, we have an entity that might trigger 100s of events per minute that we'd like to trigger flow runs for.
generally we'd just put that behind a queue and have a celery app process the different types of events, but we have a need for things like caching/more fine-grained concurrency, and don't necessarily want to implement those features ourselves (and think using prefect could be a better solution)
k
Kevin Kho
06/06/2022, 5:21 PM
I think nothing else aside from being aware that Prefect might error if it tries to deploy a flow but doesn’t have resources
j
Jai P
06/06/2022, 5:24 PM
got it, ok! if we run into problems along the way, we'll definitely reach out. thanks for the help!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.