Hi, I'm evaluating several tools to help build a team that deals with data integration.
In the toolbox, we will probably have prefect orion
I have conducted a POC where I deploy a flow on a Google Kubernetes Engine cluster which uses a Google Cloud bucket as the storage and connects to the Prefect Cloud Orion API. The flow then consumes a kafka topic hosted by Confluent Cloud and for each kafka message starts a
which is a long running kafka consumer that starts a flow when receiving a kafka message but does not require a deployment nor an agent. Based on the advices given by Anna. I think it's more suitable to my streaming use-case.