hey there. Is there any plan for a

openlineage

integration? For ref: • https://openlineage.io/ • https://github.com/OpenLineage/OpenLineage/issues/81 • there was one for Prefect v1 https://github.com/OpenLineage/OpenLineage/pull/293 CC @limx0 @Anna Geller

Ttbomk there is not at the moment, what problem are you trying to solve with Open Lineage? For transparency, I'm not a big fan of this project but I'm very interested in the underlying problems you would like to address

I’m interested on using Marquez to get dataset lineage and metadata management-discovery. Would you suggest open metadata instead? I don’t know if it integrates with Spark (scala)

Marquez doesn't give any advantage to workflow metadata that Prefect already provides in the UI. But if you are interested in metadata of your data rather than your workflow, then there are many great tools you could explore including OpenMetadata, DataHub, Atlan, Stemma and tens if not hundreds other metadata tools

Just revisiting this. I like Open Metadata, seems very interesting; didn’t play with it yet. But when it gets to a major release it seems that it will be a in a better stage. Answering your question, what I’d like is to have a single tool/framework that would be able to get data lineage from spark (ideally column level) as well as from the orchestration tool (prefect in this case; airflow has some integration with OpenLineage and Marquez).

there are like millions of lineage tools on the market, but the problem is more that everyone has a different understanding of what lineage really is

hey @Ricardo Gaspar - I'm still interested in this space, and Open Metadata looks pretty interesting. I might and have a play around with a perfect integration - would you be keen to contribute?