https://prefect.io
Join Slack
Hi there. Is there any way to have Redis act as my cache-store, instead of a filesystem? Assuming I ...
# ask-community
t
Hi there. Is there any way to have Redis act as my cache-store, instead of a filesystem? Assuming I were to create a custom block for this, how do I have the cache use Redis as a backend?
✅ 1
Just browsing the code, it seem that this should possible if I create a block which is based on 
prefect.filesystems.WriteableFileSystem
which would add the 
read-path
and 
write-path
capabilities (
get-directory
and 
put-directory
does not appear to be necessary). This can be set as storage in the deployment, and also remembering to set 
upload=False
when applying. Not a big deal as I am using 
KuberntesJob
build docker-images similar to Prefect 1. A bit hacky, but this seems feasible. Thoughts?
a
Hi! We can definitely provide a Redis storage block, but it would require a separate 
prefect-redis
collection, as we can't have that in the core Prefect repo Would you be willing to take a stab and contribute? We can ask @alex if you need early feedback but the easiest would be to contribute a collection with a storage block that has 
read-path
and 
write-path
capabilities Not sure though, whether Redis data structure would be suited for a storage block as it technically should be capable of storing entire directories Curious, given that you are using KubernetesJob as infra block, perhaps a Docker image storage is better suited for your use case? This way you wouldn't need any other system
Redis block and 
prefect-redis
collection would be valuable though regardless of whether this would be a storage block or just a custom block/task to cache something, so you can definitely take a stab to contribute the collection and see how this goes
t
sure thing, I can give this a try
👍 1
I was not aware of the new docker deployment experience - thanks for sharing 🙂 We have created our own tooling to handle deployment into our AKS-based deployments, and in the process of migrating from Prefect 1 to Orion we chose to port the docker-image creation logic from Prefect 1. This works quite well, but means that our deployments are applied without storage
a
ahh, in that case, Docker image storage is a perfect solution for your use case
I'll add to our content roadmap that we need better tutorials on using that pattern
t
awesome, thank you!
👍 1
🙏 1
a
what's your goal with this? a generic Redis client as a block? this seems valuable on its own but I don't see how this is supposed to work as a WriteableFileSystem, can you explain?
You could address the deployment problem using the Docker image storage and implement Redis block as a "pure" Block that is not a file system (because Redis is not a file system), what do you think?
t
My end goal is to have redis act as the cache. We have actually implemented this with Prefect 1 already in the form of a watermark/flow state store. The Orion cache is a lot cleaner than what we made, so switching the backend seems like the best way to achieve this
a
what do you mean by cache - a KV store?
t
essentially, yes
a
I agree that using Redis as a KV Store would be great and I would highly encourage you to contribute such a block, but this doesn't need to be a FileSystem block but rather a base Block class if you could contribute that, this would be awesome! 💯
t
wouldn't subclassing 
WritableFileSystem
essentially create a kv-store, where the key is a path?
a
it's better to create your own Block the way you need it -- KV Store stores key-value pairs, not files, so using a file system block would be confusing
t
I agree the naming is a bit confusing in this 😕
if I am looking to have this working with the Orion cache though, it seems having the capabilities 
read-path
and 
write-path
is necessary
a
I wouldn't try to use it as a cache, I thought you wanted to use it in your flow/task logic? for that, a block would make sense (e.g. to store some key-value pairs explicitly)
t
Got it, thanks for the input 🙂
In any case I have created a PR to track this conversation here: https://github.com/C4IROcean/prefect-redis/pull/1
also added some redis get/set/cmd tasks
Hi @Anna Geller - What is the process for the 
prefect-redis
repo above to become part of the official prefect packages? Do I transfer the ownership to the 
PrefectHQ
org for example?
a
@alex can answer best -- you would need to build it as a collection, but you can totally still own it as part of your GitHub account rather than PrefectHQ, the main process would be to review the code/repo settings + the release process, and list the collection in the catalog. You can also discuss this via #C0106HZ1CMS channel. Thanks for contributing!
t
thanks for your input 🙂
👍 1
🙏 1
35 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by