https://prefect.io logo
c

Clovis

12/09/2022, 10:11 AM
Hi everyone ! Is it possible to save/update only part of a
JSON
block ? I did see nothing like that in documentation. To give some context, I got multiple concurrent tasks depending on a common block and actualizing only different part of it. As I don’t want to lose data and because of the concurrency, I want to avoid loading the
JSON
block in each tasks, saving it just after with my updates and risking to overwrite it with missing value. So my question here is can I save only part of the
JSON
block ? Or maybe there is another solution (like lock or something) to prevent block data loss when it comes to async parallel task ?
m

Mason Menges

12/09/2022, 5:14 PM
Hey @Clovis what are you ultimately wanting to achieve? Normally when it comes to blocks we think of it more as some configuration for external systems that can be reused and would only really be updated when updating that configuration becomes necessary. Do you have an example of the code you're running. you could in theory load the block outside of the flow, i.e. after your imports so it can be reference by each of the tasks but depending on what you're ultimately try to achieve that may or may not be helpful.
c

Clovis

12/12/2022, 9:10 AM
Hi @Mason Menges ! Thanks for the reply 🙏 What I wanted to achieve here was to make some of my flows stateful and store some values among different runs (for instance, for incremental ETL using a time cursor). But I’m starting to think that this method could be potentially dangerous and could bring some data loss, I should probably look into something actually ACID to prevent unexpected behavior.
6 Views