https://prefect.io logo
Title
m

Mike Grabbe

01/03/2023, 7:49 PM
Is it an accepted/intended practice to have a recurring flow read/write from Prefect v2 blocks? For example, I have a python app that interacts with the Azure Graph API... this api provides a "deltatoken" output that can be used as an input to the api for subsequent requests to capture net new changes. I've used dynamodb to cache this data up to this point but Im wondering if blocks could work for this as well.
1
r

redsquare

01/03/2023, 8:02 PM
we use blocks to store checkpoints like this
m

Mike Grabbe

01/03/2023, 8:03 PM
have you found this to be a simple and reliable solution?
r

redsquare

01/03/2023, 8:03 PM
so far so good, we use a json block per flow
saves on dynamo read/writes 😁
m

Mike Grabbe

01/03/2023, 8:04 PM
I was thinking the same thing. Ok great, sounds like this is going to work then!
💯 1
r

redsquare

01/03/2023, 8:10 PM
be nice if the prefect flow had this built in so it would auto read/write a checkpoint
we are creating a decorator to save on regurgitating this functionality
m

Mike Grabbe

01/03/2023, 8:12 PM
which would automatically create/update a block per flow?
r

redsquare

01/03/2023, 8:13 PM
yeah, read a checkpoint if it exists and pass it in, store it on flow completion
I assume its a pretty common need
cc @Anna Geller @Khuyen Tran ^^ any thoughts
k

Kevin Grismore

01/03/2023, 8:32 PM
I'm doing this too. use a json block to track things like the last row number and epoch time to export rows from a source system that is updated per run of a recurring flow
👍 1
a

Andrew Huang

01/03/2023, 8:37 PM
r

redsquare

01/03/2023, 8:49 PM
yeah but you still have the reading+writing rinse & repeat work to do across all flows
z

Zanie

01/03/2023, 8:51 PM
I believe we’re going to introduce a separate concept for this, similar to the key-value store built into Prefect 1.
👀 2
Blocks are not intended for this use-case, although they can be used for it right now they’re not designed for safe read/writes from multiple actors.
r

redsquare

01/03/2023, 8:57 PM
even a block per flow?
z

Zanie

01/03/2023, 9:02 PM
If you’re not performing concurrent reads/writes, you won’t encounter any problems but it’s a pattern we’ll discourage in the future once we’ve introduced a better API for that use case.
:gratitude-thank-you: 1
a

Anna Geller

01/03/2023, 9:28 PM