https://prefect.io logo
Title
f

Filip Floegel

07/28/2022, 3:28 PM
Hi prefect evangelists :-) my name is filip and I have a problem ;-) being responsible for data governance on a real time application build… any thoughts on this in respect of prefect for managing operational data pipelines considering input & output data and some action e.g. etl in between
👋 18
j

jawnsy

07/28/2022, 3:41 PM
Welcome to the community @Filip Floegel! 👋 This seems like a great question maybe for #best-practices-coordination-plane or #prefect-community 😀
d

Dylan

07/28/2022, 3:41 PM
Hey @Filip Floegel! I found this really helpful: https://cloud.google.com/architecture/confidential-data-warehouse-blueprint
👍 4
a

Anna Geller

07/28/2022, 3:53 PM
Welcome, @Filip Floegel! 👋 a bit of self-promotion here, but you can check my post about real-time data pipelines here
:anna: 3
🔥 3
f

Filip Floegel

07/28/2022, 4:17 PM
Thank you, Great content ! Sorry for my ignorance but for basic data governance e.g. access control, encryption, data catalogue, data masking do you think the function in prefect need to get access and set encryption & get credits for decryption like Apache ranger , and predefined Schemas linked to Apache atlas catalogue s ?!
a

Anna Geller

07/28/2022, 6:23 PM
A better place for this question would be #best-practices-coordination-plane - this post is more on how to operationalize and manage real-time dataflow, but you're asking here more about data governance which is a separate problem from dataflow, and something that you may solve using e.g. our OpenMetadata integration (which will expand in the future) - OM can help you with all the areas you mentioned
👀 1
f

Filip Floegel

07/28/2022, 6:26 PM
Yes sorry :-) it will probably take some Threads till I take the right one for particular topics ;-)
j

Jeff Hale

07/29/2022, 12:09 AM
Welcome @Filip Floegel! 🚀