https://prefect.io logo
Title
v

Valentin Baert

05/19/2022, 11:54 AM
Hi, I'm evaluating several tools to help build a team that deals with data integration. In the toolbox, we will probably have prefect orion I have conducted a POC where I deploy a flow on a Google Kubernetes Engine cluster which uses a Google Cloud bucket as the storage and connects to the Prefect Cloud Orion API. The flow then consumes a kafka topic hosted by Confluent Cloud and for each kafka message starts a
@task
to process the message (just logging for this POC). I have written the steps for my POC here if it might help other people : https://gitlab.com/idkw/prefect-orion-gke-poc
❤️ 5
:thank-you: 2
:upvote: 2
🚀 5
a

Anna Geller

05/19/2022, 11:56 AM
This is a fantastic resource, can't wait to see how you progress with the real-time streaming use case - excited to assist you along the way! 👏
v

Valentin Baert

05/19/2022, 12:00 PM
Thanks, in our other thread I was wondering what should I do about this issue : https://prefect-community.slack.com/archives/CL09KU1K7/p1652949937007829?thread_ts=1651566181.776459&cid=CL09KU1K7 I don't kow if I should have a @flow wrapping the kafka infinite polling loop and then run a task for every individual message (but then how do I cleanly stop the infinite flow when I need to update the code ?) or whether there is a better way to handle event streaming. I was under the impression prefect 2.0 was brining new answers to this but I'm a bit lost here.
Should I use a sub-flow or is there a better way ?
a

Anna Geller

05/19/2022, 12:01 PM
let's continue the discussion in our previous thread, this one is only for sharing blog posts etc - but all great questions
v

Valentin Baert

05/19/2022, 12:01 PM
ok
k

Kevin Kho

05/19/2022, 1:31 PM
That’s nice! Good to see Kafka + Prefect, which was awkward in Prefect 1.0
:upvote: 1
v

Valentin Baert

05/19/2022, 1:49 PM
I have added an additional sample
prefect_2_kafka_kub_no_deployment.py
which is a long running kafka consumer that starts a flow when receiving a kafka message but does not require a deployment nor an agent. Based on the advices given by Anna. I think it's more suitable to my streaming use-case.
a

Anna Geller

05/19/2022, 1:57 PM
nice work! 🙌