Hi prefect evangelists :-) my name is filip and I have a problem ;-) being responsible for data governance on a real time application build… any thoughts on this in respect of prefect for managing operational data pipelines considering input & output data and some action e.g. etl in between
👋 18
j
jawnsy
07/28/2022, 3:41 PM
Welcome to the community @Filip Floegel! 👋
This seems like a great question maybe for #best-practices-coordination-plane or #prefect-community 😀
Welcome, @Filip Floegel! 👋 a bit of self-promotion here, but you can check my post about real-time data pipelines here
:anna: 3
🔥 3
f
Filip Floegel
07/28/2022, 4:17 PM
Thank you, Great content ! Sorry for my ignorance but for basic data governance e.g. access control, encryption, data catalogue, data masking do you think the function in prefect need to get access and set encryption & get credits for decryption like Apache ranger , and predefined Schemas linked to Apache atlas catalogue s ?!
a
Anna Geller
07/28/2022, 6:23 PM
A better place for this question would be #best-practices-coordination-plane - this post is more on how to operationalize and manage real-time dataflow, but you're asking here more about data governance which is a separate problem from dataflow, and something that you may solve using e.g. our OpenMetadata integration (which will expand in the future) - OM can help you with all the areas you mentioned
👀 1
f
Filip Floegel
07/28/2022, 6:26 PM
Yes sorry :-) it will probably take some Threads till I take the right one for particular topics ;-)