ale

    ale

    1 year ago
    Hey folks πŸ™‚ Are there any plans to add data lineage to Prefect? Maybe an integration with Marquez/Datakin?
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @ale -- I think that native data lineage in Prefect may be a bit out of scope for us. I think we're better off focusing on orchestration. People that are focusing on data lineage directly are probably going to do a better job than us so I think an integration is the way to go! We've already started on this with our Great Expectations cooperative work and I'm sure other tools can be hooked in as well πŸ™‚
    ale

    ale

    1 year ago
    Yeah I agree that an integration would be the best path! Thanks for your thoughts @Michael Adkins πŸ‘
    And I’m pretty sure that the folks that are building Marquez/Datakin/OpenLineage would be quite interested in build such an integration 😬
    Michael Adkins

    Michael Adkins

    1 year ago
    Send them our way πŸ˜„
    ale

    ale

    1 year ago
    Oh I’ve just remember that someone at Marquez/Datakin told me that they already had an initial conversation with you guys! Julien from Datakin maybe?
    Jeremiah

    Jeremiah

    1 year ago
    :yes: we’re very excited to see the progress on Datakin/OpenLineage.