Hello guys, I have a data source for real-time. And comes 10k data every minute. I want to process these data with dag. Do you think prefect is a good choice?
a
Anna Geller
05/05/2022, 5:04 PM
Definitely! Check this blog post showing an example of a streaming use case in Prefect 2.0
upvote 1
m
massumo
05/05/2022, 5:09 PM
What about the performance? Could i handle?
k
Kevin Kho
05/05/2022, 5:18 PM
With 2.0 there is less spin up time. You could have a long running flow that keeps submitting tasks and it might be lightweight enough.