Is it an accepted/intended practice to have a recu...
# prefect-community
m
Is it an accepted/intended practice to have a recurring flow read/write from Prefect v2 blocks? For example, I have a python app that interacts with the Azure Graph API... this api provides a "deltatoken" output that can be used as an input to the api for subsequent requests to capture net new changes. I've used dynamodb to cache this data up to this point but Im wondering if blocks could work for this as well.
1
r
we use blocks to store checkpoints like this
m
have you found this to be a simple and reliable solution?
r
so far so good, we use a json block per flow
saves on dynamo read/writes 😁
m
I was thinking the same thing. Ok great, sounds like this is going to work then!
💯 1
r
be nice if the prefect flow had this built in so it would auto read/write a checkpoint
we are creating a decorator to save on regurgitating this functionality
m
which would automatically create/update a block per flow?
r
yeah, read a checkpoint if it exists and pass it in, store it on flow completion
I assume its a pretty common need
cc @Anna Geller @Khuyen Tran ^^ any thoughts
k
I'm doing this too. use a json block to track things like the last row number and epoch time to export rows from a source system that is updated per run of a recurring flow
👍 1
a
r
yeah but you still have the reading+writing rinse & repeat work to do across all flows
z
I believe we’re going to introduce a separate concept for this, similar to the key-value store built into Prefect 1.
👀 2
Blocks are not intended for this use-case, although they can be used for it right now they’re not designed for safe read/writes from multiple actors.
r
even a block per flow?
z
If you’re not performing concurrent reads/writes, you won’t encounter any problems but it’s a pattern we’ll discourage in the future once we’ve introduced a better API for that use case.
gratitude thank you 1
a