Hey all, been spending the past week or so playing around with Prefect and Dagster to see which one better fits my team's use cases. We're heavily leaning towards Prefect, but I wanted to see if there's an equivalent to the
AssetMaterialization
that Dagster offers to allow us to surface metadata about a Flow or Task. I feel like this could be very helpful for our support teams who are often the first line of defense when a data pipeline misbehaves. Thank you!
c
Chris White
09/11/2024, 3:04 PM
Hey William! What sort of metadata would you hope to be able to surface?
w
William Hom
09/11/2024, 4:09 PM
Thanks for the response! Hard to say, really, since it'll depend her Pipeline. But for example, if I have a Task that hits an API endpoint and returns N records, I would like a way to surface that information to someone without them having to comb through the logs. In Dagster, someone can click on an "Asset" and that information is readily available to them in the UI.
c
Chris White
09/11/2024, 4:18 PM
Gotcha, so for UI visibility I'd recommend starting with a Prefect "artifact", and in this case maybe a simple markdown artifact: https://docs.prefect.io/3.0/develop/artifacts#create-markdown-artifacts (the code example here should also work in 2.x if you're using that).
Definitely curious to hear if that satisfies your needs or not, this is something I'm particularly interested in
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.