Yeah, that is what I was trying to remember, but I dont think it is what I want. Trying to work around BQ rate limits on writing some context (and other) metadata to the db
Tim Enders
01/20/2022, 7:02 PM
Wait, is there a doc/definition for the GraphQL API somewhere?
k
Kevin Kho
01/20/2022, 7:05 PM
Oh I don’t think there is room for that to store metadata in the Prefect database. The documentation would be in the
Interactive API
tab of the UI
Kevin Kho
01/20/2022, 7:06 PM
How big is this metadata?
t
Tim Enders
01/21/2022, 2:26 PM
It is some pieces of the context and then some metrics from the run. Record counts, page status on APIs
k
Kevin Kho
01/21/2022, 2:37 PM
I think conceptually using a task to output a result would make the most sense. We don’t have any place in the database to insert information.
t
Tim Enders
01/21/2022, 2:41 PM
Yeah, I just need to write data form multiple tasks to one metadata table. And I don't want to stream the data in if I don't have to. Thinking a Flow Run ID named file will be unique enough for me to stash it all and write at the end of the Flow.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.