https://prefect.io logo
Title
t

Thomas Fredriksen

10/07/2022, 8:37 AM
Hi there. Is there any way to have Redis act as my cache-store, instead of a filesystem? Assuming I were to create a custom block for this, how do I have the cache use Redis as a backend?
1
Just browsing the code, it seem that this should possible if I create a block which is based on
prefect.filesystems.WriteableFileSystem
which would add the
read-path
and
write-path
capabilities (
get-directory
and
put-directory
does not appear to be necessary). This can be set as storage in the deployment, and also remembering to set
upload=False
when applying. Not a big deal as I am using
KuberntesJob
build docker-images similar to Prefect 1. A bit hacky, but this seems feasible. Thoughts?
a

Anna Geller

10/07/2022, 12:13 PM
Hi! We can definitely provide a Redis storage block, but it would require a separate
prefect-redis
collection, as we can't have that in the core Prefect repo Would you be willing to take a stab and contribute? We can ask @alex if you need early feedback but the easiest would be to contribute a collection with a storage block that has
read-path
and
write-path
capabilities Not sure though, whether Redis data structure would be suited for a storage block as it technically should be capable of storing entire directories Curious, given that you are using KubernetesJob as infra block, perhaps a Docker image storage is better suited for your use case? This way you wouldn't need any other system
Redis block and
prefect-redis
collection would be valuable though regardless of whether this would be a storage block or just a custom block/task to cache something, so you can definitely take a stab to contribute the collection and see how this goes
t

Thomas Fredriksen

10/07/2022, 12:22 PM
sure thing, I can give this a try
👍 1
I was not aware of the new docker deployment experience - thanks for sharing 🙂 We have created our own tooling to handle deployment into our AKS-based deployments, and in the process of migrating from Prefect 1 to Orion we chose to port the docker-image creation logic from Prefect 1. This works quite well, but means that our deployments are applied without storage
a

Anna Geller

10/07/2022, 12:28 PM
ahh, in that case, Docker image storage is a perfect solution for your use case
I'll add to our content roadmap that we need better tutorials on using that pattern
t

Thomas Fredriksen

10/07/2022, 12:30 PM
awesome, thank you!
👍 1
Quick'n'dirty rapper around the
redis-py
client: https://github.com/C4IROcean/prefect-redis/blob/master/prefect_redis/blocks.py
:thank-you: 1
a

Anna Geller

10/07/2022, 3:21 PM
what's your goal with this? a generic Redis client as a block? this seems valuable on its own but I don't see how this is supposed to work as a WriteableFileSystem, can you explain?
You could address the deployment problem using the Docker image storage and implement Redis block as a "pure" Block that is not a file system (because Redis is not a file system), what do you think?
t

Thomas Fredriksen

10/07/2022, 5:52 PM
My end goal is to have redis act as the cache. We have actually implemented this with Prefect 1 already in the form of a watermark/flow state store. The Orion cache is a lot cleaner than what we made, so switching the backend seems like the best way to achieve this
a

Anna Geller

10/07/2022, 5:55 PM
what do you mean by cache - a KV store?
t

Thomas Fredriksen

10/07/2022, 5:56 PM
essentially, yes
a

Anna Geller

10/07/2022, 5:56 PM
I agree that using Redis as a KV Store would be great and I would highly encourage you to contribute such a block, but this doesn't need to be a FileSystem block but rather a base Block class if you could contribute that, this would be awesome! 💯
t

Thomas Fredriksen

10/07/2022, 5:57 PM
wouldn't subclassing
WritableFileSystem
essentially create a kv-store, where the key is a path?
a

Anna Geller

10/07/2022, 5:59 PM
it's better to create your own Block the way you need it -- KV Store stores key-value pairs, not files, so using a file system block would be confusing
t

Thomas Fredriksen

10/07/2022, 6:00 PM
I agree the naming is a bit confusing in this 😕
if I am looking to have this working with the Orion cache though, it seems having the capabilities
read-path
and
write-path
is necessary
a

Anna Geller

10/07/2022, 7:08 PM
I wouldn't try to use it as a cache, I thought you wanted to use it in your flow/task logic? for that, a block would make sense (e.g. to store some key-value pairs explicitly)
t

Thomas Fredriksen

10/07/2022, 7:11 PM
Got it, thanks for the input 🙂
In any case I have created a PR to track this conversation here: https://github.com/C4IROcean/prefect-redis/pull/1
also added some redis get/set/cmd tasks
Hi @Anna Geller - What is the process for the
prefect-redis
repo above to become part of the official prefect packages? Do I transfer the ownership to the
PrefectHQ
org for example?
a

Anna Geller

10/10/2022, 12:16 PM
@alex can answer best -- you would need to build it as a collection, but you can totally still own it as part of your GitHub account rather than PrefectHQ, the main process would be to review the code/repo settings + the release process, and list the collection in the catalog. You can also discuss this via #prefect-contributors channel. Thanks for contributing!
t

Thomas Fredriksen

10/10/2022, 12:17 PM
thanks for your input 🙂
👍 1
:thank-you: 1