https://prefect.io logo
Title
m

massumo

05/05/2022, 4:58 PM
Hello guys, I have a data source for real-time. And comes 10k data every minute. I want to process these data with dag. Do you think prefect is a good choice?
a

Anna Geller

05/05/2022, 5:04 PM
Definitely! Check this blog post showing an example of a streaming use case in Prefect 2.0
:upvote: 1
m

massumo

05/05/2022, 5:09 PM
What about the performance? Could i handle?
k

Kevin Kho

05/05/2022, 5:18 PM
With 2.0 there is less spin up time. You could have a long running flow that keeps submitting tasks and it might be lightweight enough.