https://prefect.io logo
j

Jeff Yun

09/30/2019, 6:47 PM
Related to my above question, how do I access the value of a parameter? For example, I want to use
flattened_lists = itertools.chain(*nested_lists_param)
but can't iterate through the Parameter itself.
c

Chris White

09/30/2019, 6:49 PM
Hi @Jeff Yun - remember that Prefect builds a deferred computation graph, so no code is executed until the Flow is actually run (which also means the values of tasks / parameters are only available at run-time). That being said, you should be able to do:
Copy code
@task(name='create-product')
def create_product(params1, params2):
   return list(itertools.product(params1, params2))
and then map over the output of
create_product
j

Jeff Yun

09/30/2019, 6:52 PM
Thanks for your response! It might be too brute-force since I don't need the product list itself. Is there a more memory-efficient way than to create and pass back the entire product list (params1 and params2 could have length of 1000+)? I'm pretty new to Prefect and not sure about memory management in tasks.
c

Chris White

09/30/2019, 6:56 PM
Anytime! It’s an interesting question; unfortunately I can’t think of a more memory efficient design at the moment - Prefect was designed to run in a distributed context, so all outputs are required to be serializable - this means that iterators aren’t supported as outputs at this time. I’ll give it some thought though and maybe there’s a more clever design lurking in here
👍 1
j

Jeff Yun

09/30/2019, 7:00 PM
So I can do something like?
Copy code
@task
do_something = lambda p1, p2: one_thing(p1, p2)

prod = create_product(p1, p2) 
result = do_something.map(prod)
Great, thanks!