https://prefect.io logo
a

ale

04/29/2021, 3:30 PM
Hey folks πŸ™‚ Are there any plans to add data lineage to Prefect? Maybe an integration with Marquez/Datakin?
z

Zanie

04/29/2021, 3:41 PM
Hey @ale -- I think that native data lineage in Prefect may be a bit out of scope for us. I think we're better off focusing on orchestration. People that are focusing on data lineage directly are probably going to do a better job than us so I think an integration is the way to go! We've already started on this with our Great Expectations cooperative work and I'm sure other tools can be hooked in as well πŸ™‚
a

ale

04/29/2021, 3:43 PM
Yeah I agree that an integration would be the best path! Thanks for your thoughts @Zanie πŸ‘
And I’m pretty sure that the folks that are building Marquez/Datakin/OpenLineage would be quite interested in build such an integration 😬
z

Zanie

04/29/2021, 3:54 PM
Send them our way πŸ˜„
a

ale

04/29/2021, 4:10 PM
Oh I’ve just remember that someone at Marquez/Datakin told me that they already had an initial conversation with you guys! Julien from Datakin maybe?
j

Jeremiah

04/30/2021, 2:21 PM
πŸ‘Œ we’re very excited to see the progress on Datakin/OpenLineage.
πŸ‘ 1
4 Views