Hello, I am learning Prefect and I am trying to make a flow inside a flow which takes a value/parameter from the above flow.
I got a database file with a flow which fetches a dataframe and I want to pass that data to a new flow in another file to work with the data.
I came across this doc https://docs.prefect.io/core/idioms/flow-to-flow.html, but I am still not sure how you could share data between flows.
Copy code
flow_database = StartFlowRun(flow_name="database", project_name="test", wait=True)
flow_transform = StartFlowRun(flow_name="transform", project_name="test", wait=True)
with Flow("parent-flow") as flow:
data = flow_database() #get a dataframe from flow
flow_transform(data) # take dataframe and use it another flow
How should I go about this?
a
ale
04/13/2021, 3:01 PM
Hey @Jonas
If you want to pass data between different flows, then you have to use some sort of external storage (object storage, database, files, etc).
Thanks for the reponse guys. The thread was usefull to read, but I guess I have to wait before this idea is implemented. I will try looking at external storage
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.