Thomas Fredriksen
10/07/2022, 8:37 AMprefect.filesystems.WriteableFileSystem
which would add the read-path
and write-path
capabilities (get-directory
and put-directory
does not appear to be necessary).
This can be set as storage in the deployment, and also remembering to set upload=False
when applying. Not a big deal as I am using KuberntesJob
build docker-images similar to Prefect 1.
A bit hacky, but this seems feasible. Thoughts?Anna Geller
10/07/2022, 12:13 PMprefect-redis
collection, as we can't have that in the core Prefect repo
Would you be willing to take a stab and contribute?
We can ask @alex if you need early feedback but the easiest would be to contribute a collection with a storage block that has read-path
and write-path
capabilities
Not sure though, whether Redis data structure would be suited for a storage block as it technically should be capable of storing entire directories
Curious, given that you are using KubernetesJob as infra block, perhaps a Docker image storage is better suited for your use case? This way you wouldn't need any other systemprefect-redis
collection would be valuable though regardless of whether this would be a storage block or just a custom block/task to cache something, so you can definitely take a stab to contribute the collection and see how this goesThomas Fredriksen
10/07/2022, 12:22 PMAnna Geller
10/07/2022, 12:28 PMThomas Fredriksen
10/07/2022, 12:30 PMredis-py
client: https://github.com/C4IROcean/prefect-redis/blob/master/prefect_redis/blocks.pyAnna Geller
10/07/2022, 3:21 PMThomas Fredriksen
10/07/2022, 5:52 PMAnna Geller
10/07/2022, 5:55 PMThomas Fredriksen
10/07/2022, 5:56 PMAnna Geller
10/07/2022, 5:56 PMThomas Fredriksen
10/07/2022, 5:57 PMWritableFileSystem
essentially create a kv-store, where the key is a path?Anna Geller
10/07/2022, 5:59 PMThomas Fredriksen
10/07/2022, 6:00 PMread-path
and write-path
is necessaryAnna Geller
10/07/2022, 7:08 PMThomas Fredriksen
10/07/2022, 7:11 PMprefect-redis
repo above to become part of the official prefect packages? Do I transfer the ownership to the PrefectHQ
org for example?Anna Geller
10/10/2022, 12:16 PMThomas Fredriksen
10/10/2022, 12:17 PM