https://prefect.io
Join Slack
Hi, I was trying to find some information on using redis as a store to persist data between task run...
# ask-community
s
Hi, I was trying to find some information on using redis as a store to persist data between task runs (or flow runs). Is there any? Thx!
Or any other means to persist output as input to a new flow/task run in memory.
k
I am a bit confused why it wouldn’t just work if you pass it directly?
Copy code
@flow
def myflow():
    a = task_one()
    b = task_two(a)
and it should work in memory For across flow runs, if you use a subflow, it’s also passed. Are you seeing something different?
s
Hi @Kevin Kho , sorry for the delayed response. I had something like this in mind:
Copy code
From collections import deque
from random import random

SOME_GLOBAL_STORE = deque(10)

@task
def my_task(new_item):
    SOME_GLOBAL_STORE.append(left(new_item)
    return sum(SOME_GLOBAL_STORE)


@flow
def my_flow()
    new_item = random()
    return my_task(new_item)
k
That doc is 1.0 though
s
But I haven't seen some solution for using redis as a store
Is something like that available for 2.0 or somewhere on the roadmap?
k
Actually 1.0 has a KV Store which was a cache for something you needed across flow runs. No implementation yet for 2.0
s
Does that go through the filesystem?
k
Task returns are currently persisted in 2.0 based on default storage, but the interface will change pretty soon. I think Flow returns might be as well. When that gets put in place, then I guess we can explicitly talk about Redis.
The Prefect 1.0 KV Store does not go through the file system. It gets inserted in the database
Just to be clear, you are asking for some sort of persistent store to be used after the Flow run or across Flow Runs right?
s
We would prefer some localised solution in memory, to ensure efficient performance.
Across flow runs.
So that we're able to accumulate some sort of moving average for each flow run.
k
In-memory across flow runs I don’t think is doable just because they happen on different compute but we can make a feature request to support Redis as an intermediate store. But I can see this being kind of tricky because the design is really for filesystem. For now, I can open a feature request to surface the idea but this is honestly a while out if ever.
👍 1
@Marvin open “Orion Idea - Support Redis as An Intermediate Store Between Flows”
m
s
Thanks!
If we'd want to build this I guess your team is open to review the PR?
k
Yes we review the open source PRs. I’d gladly volunteer a colleague
🚀 1
s
Nothing planned yet but just checking:)
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by