https://prefect.io logo
t

Tenzin Choedak

10/05/2020, 6:38 PM
Hi prefect community and prefect team 👋 Is there any desire for a builtin prefect task for kafka flows? At my org, within the data engineering team we hardly ever run live producers and consumers - those are typically maintained and developed by the application team who need that sub-second responsiveness to events. However, we do own batch consumers and producers - either to pickup a batch of new events from the application stack or to push a batch of updates to the application stack. It seems possible to at least design a batch consume + batch produce task for a kafka integration within prefect. How do others and the prefect team feel about that?
k

Kyle Moon-Wright

10/05/2020, 7:14 PM
Hey @Tenzin Choedak, We'd love to see a Kafka task! The Task library is largely driven by use cases and open-source initiatives from the community, so this is the best place to find motivation for integrations. I would certainly love to see a Kafka task, in addition to user feedback here, a Github issue could be a great starting point to collect data on use cases and discuss with the Core developers.
I know there are Kafka discussions here in #prefect-community, but I haven't been able to search - Slack is a bit unstable today.
t

Tenzin Choedak

10/05/2020, 7:16 PM
@Kyle Moon-Wright thanks for that suggestion - I’ll throw an issue open to help centralize the discussion
Yeah lol Slack is a bit wonky today!
🙃 1
@Kyle Moon-Wright I think I went through a new github flow for creating issues, it actually lead me to a new page called
discussions
. https://github.com/PrefectHQ/prefect/discussions/3442 Is this the right place to propose a new task? Or should I just copy/paste into a new
issues
tab?
k

Kyle Moon-Wright

10/06/2020, 1:03 AM
This is great, thank you for these details! Discussions works as well. Nice job on this!