👋 hullo! i'm curious, if i want to use prefect 2.0 for workflows that i'd typically do with something like celery (e.g. enabling an event-driven architecture vs. more ETL-y type workloads), is that a bad idea?
06/06/2022, 5:10 PM
No it’s not and it’s something we think 2.0 can handle. The only thing is that Prefect is not resource-aware so you need to explicitly set the flow concurrency yourself for you agent. Does that make sense?
You can use the API to create a deployment
06/06/2022, 5:13 PM
hmm, i think that makes sense! and i'd set the flow concurrency in the deployment spec?
oh, or is it
work queue concurrency
that i'd have to set
06/06/2022, 5:14 PM
06/06/2022, 5:17 PM
awesome! is there anything else you'd suggest we look out for? to paint a picture, we have an entity that might trigger 100s of events per minute that we'd like to trigger flow runs for.
generally we'd just put that behind a queue and have a celery app process the different types of events, but we have a need for things like caching/more fine-grained concurrency, and don't necessarily want to implement those features ourselves (and think using prefect could be a better solution)
06/06/2022, 5:21 PM
I think nothing else aside from being aware that Prefect might error if it tries to deploy a flow but doesn’t have resources
06/06/2022, 5:24 PM
got it, ok! if we run into problems along the way, we'll definitely reach out. thanks for the help!