Hi prefect evangelists :-) my name is filip and I have a problem ;-) being responsible for data governance on a real time application build… any thoughts on this in respect of prefect for managing operational data pipelines considering input & output data and some action e.g. etl in between

Welcome to the community @Filip Floegel! 👋 This seems like a great question maybe for #best-practices-coordination-plane or #prefect-community 😀

Hey @Filip Floegel! I found this really helpful: https://cloud.google.com/architecture/confidential-data-warehouse-blueprint

Welcome, @Filip Floegel! 👋 a bit of self-promotion here, but you can check my post about real-time data pipelines here

Thank you, Great content ! Sorry for my ignorance but for basic data governance e.g. access control, encryption, data catalogue, data masking do you think the function in prefect need to get access and set encryption & get credits for decryption like Apache ranger , and predefined Schemas linked to Apache atlas catalogue s ?!

A better place for this question would be #best-practices-coordination-plane - this post is more on how to operationalize and manage real-time dataflow, but you're asking here more about data governance which is a separate problem from dataflow, and something that you may solve using e.g. our OpenMetadata integration (which will expand in the future) - OM can help you with all the areas you mentioned

Yes sorry :-) it will probably take some Threads till I take the right one for particular topics ;-)

Welcome @Filip Floegel! 🚀