https://prefect.io logo
Title
o

Ofir

03/12/2023, 4:52 PM
Is it possible to persistently annotate/attach metadata to a successful deployment run in Prefect with metadata or should I use storage for that? We use Prefect to asynchronously launch a short-lived workflow (<1min) from a Node.js backend. This allows us to free the Node.js backend from being blocked. The output of this workflow is a simple JSON with an array of key-value pairs. We would like to be able to return the result of this JSON in the simplest possible way. What’s the best practice here? AFAICT there are 3 options: 1. Persist the result into a database from the workflow 2. Persist the result into a storage 3. Annotate the deployment run / attach metadata to the deployment run using Prefect SDK If option (3) exists it would be great, less responsibilities on the application side of things to setup the database, persist the data and then apply garbage collection to stale data (e.g. when deployments get deleted, we need to do a cascaded delete where the JSON body also needs to be wiped out). Your thoughts? Thanks!
a

Austin Weisgrau

03/12/2023, 5:34 PM
You can also attach a callback function to the flow run, so after it finishes you could have it post data to a node endpoint
o

Ofir

03/12/2023, 5:34 PM
Sure but what happens if this node dies?
I would like to capitalize on Prefect’s persistence in some shape or form.
👍 1
a

Austin Weisgrau

03/12/2023, 5:35 PM
Ah yeah that Makes sense
🙏 1
j

Jeff Hale

03/13/2023, 12:29 AM
There’s works being done on this front. PRs you can follow here: https://github.com/PrefectHQ/prefect/pull/8645 https://github.com/PrefectHQ/prefect/pull/8759
o

Ofir

03/13/2023, 10:59 AM
Thanks Jeff, what an interesting timing!
@Jeff Hale what’s the ETA for merging them and are they planned to be as part of an upcoming Prefect release?
j

Jeff Hale

03/13/2023, 1:01 PM
Hi Ofir. I know they are being actively worked on and will be added to a release as soon as they are ready. 🙂
o

Ofir

03/13/2023, 1:01 PM
Gotcha thanks!
👍 1