https://prefect.io logo
Title
s

Sander

05/24/2022, 8:45 PM
Hi, I was trying to find some information on using redis as a store to persist data between task runs (or flow runs). Is there any? Thx!
Or any other means to persist output as input to a new flow/task run in memory.
k

Kevin Kho

05/24/2022, 8:52 PM
I am a bit confused why it wouldn’t just work if you pass it directly?
@flow
def myflow():
    a = task_one()
    b = task_two(a)
and it should work in memory For across flow runs, if you use a subflow, it’s also passed. Are you seeing something different?
s

Sander

05/26/2022, 6:53 PM
Hi @Kevin Kho , sorry for the delayed response. I had something like this in mind:
From collections import deque
from random import random

SOME_GLOBAL_STORE = deque(10)

@task
def my_task(new_item):
    SOME_GLOBAL_STORE.append(left(new_item)
    return sum(SOME_GLOBAL_STORE)


@flow
def my_flow()
    new_item = random()
    return my_task(new_item)
k

Kevin Kho

05/26/2022, 6:54 PM
That doc is 1.0 though
s

Sander

05/26/2022, 6:54 PM
But I haven't seen some solution for using redis as a store
Is something like that available for 2.0 or somewhere on the roadmap?
k

Kevin Kho

05/26/2022, 6:55 PM
Actually 1.0 has a KV Store which was a cache for something you needed across flow runs. No implementation yet for 2.0
s

Sander

05/26/2022, 6:56 PM
Does that go through the filesystem?
k

Kevin Kho

05/26/2022, 6:56 PM
Task returns are currently persisted in 2.0 based on default storage, but the interface will change pretty soon. I think Flow returns might be as well. When that gets put in place, then I guess we can explicitly talk about Redis.
The Prefect 1.0 KV Store does not go through the file system. It gets inserted in the database
Just to be clear, you are asking for some sort of persistent store to be used after the Flow run or across Flow Runs right?
s

Sander

05/26/2022, 6:57 PM
We would prefer some localised solution in memory, to ensure efficient performance.
Across flow runs.
So that we're able to accumulate some sort of moving average for each flow run.
k

Kevin Kho

05/26/2022, 7:10 PM
In-memory across flow runs I don’t think is doable just because they happen on different compute but we can make a feature request to support Redis as an intermediate store. But I can see this being kind of tricky because the design is really for filesystem. For now, I can open a feature request to surface the idea but this is honestly a while out if ever.
👍 1
@Marvin open “Orion Idea - Support Redis as An Intermediate Store Between Flows”
s

Sander

05/26/2022, 7:15 PM
Thanks!
If we'd want to build this I guess your team is open to review the PR?
k

Kevin Kho

05/26/2022, 7:17 PM
Yes we review the open source PRs. I’d gladly volunteer a colleague
🚀 1
s

Sander

05/26/2022, 7:18 PM
Nothing planned yet but just checking:)