Hi there. Is there any way to have Redis act as my cache-store, instead of a filesystem? Assuming I were to create a custom block for this, how do I have the cache use Redis as a backend?

Hi! We can definitely provide a Redis storage block, but it would require a separate

prefect-redis

collection, as we can't have that in the core Prefect repo Would you be willing to take a stab and contribute? We can ask @alex if you need early feedback but the easiest would be to contribute a collection with a storage block that has

read-path

and

write-path

capabilities Not sure though, whether Redis data structure would be suited for a storage block as it technically should be capable of storing entire directories Curious, given that you are using KubernetesJob as infra block, perhaps a Docker image storage is better suited for your use case? This way you wouldn't need any other system

sure thing, I can give this a try

ahh, in that case, Docker image storage is a perfect solution for your use case

awesome, thank you!

what's your goal with this? a generic Redis client as a block? this seems valuable on its own but I don't see how this is supposed to work as a WriteableFileSystem, can you explain?

My end goal is to have redis act as the cache. We have actually implemented this with Prefect 1 already in the form of a watermark/flow state store. The Orion cache is a lot cleaner than what we made, so switching the backend seems like the best way to achieve this

what do you mean by cache - a KV store?

essentially, yes

I agree that using Redis as a KV Store would be great and I would highly encourage you to contribute such a block, but this doesn't need to be a FileSystem block but rather a base Block class if you could contribute that, this would be awesome! 💯

wouldn't subclassing

WritableFileSystem

essentially create a kv-store, where the key is a path?

it's better to create your own Block the way you need it -- KV Store stores key-value pairs, not files, so using a file system block would be confusing

I agree the naming is a bit confusing in this 😕

I wouldn't try to use it as a cache, I thought you wanted to use it in your flow/task logic? for that, a block would make sense (e.g. to store some key-value pairs explicitly)

Got it, thanks for the input 🙂

@alex can answer best -- you would need to build it as a collection, but you can totally still own it as part of your GitHub account rather than PrefectHQ, the main process would be to review the code/repo settings + the release process, and list the collection in the catalog. You can also discuss this via #prefect-contributors channel. Thanks for contributing!

thanks for your input 🙂