Hi, I was wondering if anyone has tried using Redis as the storage backend for orion (prefect 2)? I see there is no built-in support to it. https://orion-docs.prefect.io/concepts/storage/ The closest we get is KV Server Storage. To use redis, does that mean I need to expose a HTTP API in between Orion and the redis server?

Redis support for storage is definitely possible! It is on our roadmap but if you want to implement it yourself, the KV storage is a good model.

Thanks you for the prompt reply! I am glad to hear that redis support is on the way. My project involves a large data processing pipeline that is currently built with redis + celery. We use redis + rejson (https://oss.redis.com/redisjson/) as our kv store for keeping track of the celery tasks, celery beat schedules, and intemediate data product cache. I see that the KV storage expect JSON as data blob, are the data blob always queried altogether on the KV server, or the internal calls to the KV server API involves querying a subtree of the JSON data? I am asking this because the JSON blob we stored in the redis sometimes is quite large and the nice thing about REJSON is that it can query deeply nested values in a large JSON blob.

We use redis + rejson (https://oss.redis.com/redisjson/) as our kv store for keeping track of the celery tasks

Given that Prefect 2.0 ships with work queues, it's possible that you no longer need to use Redis for that. Queued runs are stored in the backend.

I see what you mean. With Redis + Celery, we use Redis to do ALL the data management, including the queue and task metadata and task results. With prefect2, like you said, the queue, flow, task and task scheduling is builtin (stored in the sqlite or postgres database), so there is no need to use redis for those any more. However, our use case still needs redis as the data store, to store application state, intermediate data product. What our celery tasks do is to create/update these data products stored in the redis in realtime, trigged by external events (e.g, a new observation request, new data files appearing on the filesystem, etc)

Thanks for explaining, sounds like a perfect use case for Prefect 2.0!

prefect2 store is exactly to do this, managing task results.

not exactly. Storage is to store flow code. Task run result storage is a bit different and this is something we are still working on, so it’s best if you watch our #CKNSX5WG3 for new releases and wait a bit until this feature is out.

I see. I'll keep an eye for announcement. Thank you for the answers. In this case, I guess at the least, we can bypass the prefect store mechanism for storing result, by always returning None, or some state code (success/fail) in all the tasks, and in the task body, we do whatever we want to talk the redis database and update the application state