Rob Fowler

    Rob Fowler

    1 year ago
    any one already made a redis Result class? I am using docker storage and I think maybe LocalResults to a nfs/gfs mount or make a redis one?
    Well that works really well. I am surprised that one is not in the product. Specially as an option for Docker storage. You want it @Chris White ?
    Dylan

    Dylan

    1 year ago
    @Rob Fowler We’d love it if you contributed a
    RedisResult
    to Prefect!
    Chris White

    Chris White

    1 year ago
    Yup, agreed — although I am very curious about the use case here; results are generally intended for “long form” storage (e.g., restart a flow run from failure after a day) or when infrastructure messes up, and a redis result won’t be able to offer that sort of functionality.
    Rob Fowler

    Rob Fowler

    1 year ago
    Finished it. The main use case for me is with Docker storage. With the redis helm chart you can have a very reliable/HA redis cluster that is still very fast. For me, I don't need the high availability, I want to be able to restart my workflows within a few minutes to hours. They are short but complex. I expose the set() ex expiry parameter so I set keys to expire after one day. In truth, outside of a k8s cluster I have had redis instances running for literally years. The code looks like a combo of the other Result handlers, trivial. I'll make a PR against prefect. I have been running it a few days just as an extension module.
    That said, it's not an unlimited storage option like S3, GCS and azure blobs. It is just a replacement if you are vibed out by LocalStorage.
    Chris White

    Chris White

    1 year ago
    Gotcha, makes sense!
    Rob Fowler

    Rob Fowler

    1 year ago
    I started a PR but please ignore it until I make a fix or two. The cloud serialisers are byte based but the redis interface is strings. I need to look at redis a bit more or choose another serialiser. I only found this because mypy complained. Which is good.
    Dylan

    Dylan

    1 year ago
    +1 to mypy
    Rob Fowler

    Rob Fowler

    1 year ago
    Trying to finish it off but I don't seem to be able to get mypy to know the configuration argument is filled in and impossible for it not to be.
    https://github.com/PrefectHQ/prefect/blob/19f6adc8844a7d249ef49255ff6c89f6db038a5a/src/prefect/engine/results/redis_result.py#L36 src/prefect/engine/results/redis_result.py:36: error: Argument 1 to "from_url" of "Redis" has incompatible type "Optional[str]"; expected "str"
    Open to suggestions. For some reason the azure blob code that does the same thing passes.
    fix that bit, next is to look at the redis.get returned types and I'm done. I know redis get and set are binary strings but I think they are stringed somehow.