Thomas Fredriksen
10/07/2022, 8:37 AMThomas Fredriksen
10/07/2022, 8:48 AMprefect.filesystems.WriteableFileSystem
which would add the read-path
and write-path
capabilities (get-directory
and put-directory
does not appear to be necessary).
This can be set as storage in the deployment, and also remembering to set upload=False
when applying. Not a big deal as I am using KuberntesJob
build docker-images similar to Prefect 1.
A bit hacky, but this seems feasible. Thoughts?Anna Geller
prefect-redis
collection, as we can't have that in the core Prefect repo
Would you be willing to take a stab and contribute?
We can ask @alex if you need early feedback but the easiest would be to contribute a collection with a storage block that has read-path
and write-path
capabilities
Not sure though, whether Redis data structure would be suited for a storage block as it technically should be capable of storing entire directories
Curious, given that you are using KubernetesJob as infra block, perhaps a Docker image storage is better suited for your use case? This way you wouldn't need any other systemAnna Geller
prefect-redis
collection would be valuable though regardless of whether this would be a storage block or just a custom block/task to cache something, so you can definitely take a stab to contribute the collection and see how this goesThomas Fredriksen
10/07/2022, 12:22 PMThomas Fredriksen
10/07/2022, 12:25 PMAnna Geller
Anna Geller
Thomas Fredriksen
10/07/2022, 12:30 PMThomas Fredriksen
10/07/2022, 1:50 PMredis-py
client: https://github.com/C4IROcean/prefect-redis/blob/master/prefect_redis/blocks.pyAnna Geller
Anna Geller
Thomas Fredriksen
10/07/2022, 5:52 PMAnna Geller
Thomas Fredriksen
10/07/2022, 5:56 PMAnna Geller
Thomas Fredriksen
10/07/2022, 5:57 PMWritableFileSystem
essentially create a kv-store, where the key is a path?Anna Geller
Thomas Fredriksen
10/07/2022, 6:00 PMThomas Fredriksen
10/07/2022, 6:02 PMread-path
and write-path
is necessaryThomas Fredriksen
10/07/2022, 6:05 PMAnna Geller
Thomas Fredriksen
10/07/2022, 7:11 PMThomas Fredriksen
10/07/2022, 7:12 PMThomas Fredriksen
10/07/2022, 7:12 PMThomas Fredriksen
10/10/2022, 8:55 AMprefect-redis
repo above to become part of the official prefect packages? Do I transfer the ownership to the PrefectHQ
org for example?Anna Geller
Thomas Fredriksen
10/10/2022, 12:17 PM