Hi all,
Playing with Prefect for some time now and I love it so far.
I have a question regarding passing data in a flow to all tasks. Let's say I have a task A that returns some data (I know it returns actually a task object) that I want to use through all future tasks. I can pass the result as a parameter to all of the tasks but if I have this situation with more than 5 parameters (the return data from 5 tasks A-E) it's getting ugly and hard to maintain. Is there a way to set the values as global somehow (I've tried using context but without success) like in an OOP approach?
a
Anna Geller
12/10/2021, 10:02 AM
There is no way to share task return values globally.
But if all you do is passing specifically Parameter values, then you could redesign your Parameter to make it a dictionary - instead of having 5 Parameter tasks you would have only one that contains all key value pairs in a single dictionary.
Anna Geller
12/10/2021, 10:02 AM
cc @Cristian Toma
c
Cristian Toma
12/10/2021, 10:44 AM
@Anna Geller like this?
parameter = {
"a" = taskA,
"b" = taskB
}
taskC(parameter)
a
Anna Geller
12/10/2021, 10:54 AM
I thought you meant Prefect Parameter when you said parameter? 😄 but both would work provided they are passed properly as data dependencies
Copy code
from prefect import task, Parameter, Flow
@task
def do_sth_based_on_params(x):
# do sth with x
pass
with Flow("param") as flow:
par = Parameter("param_name", default=dict(key1="value1", key2="value2", key3="value3"))
do_sth_based_on_params(par)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.