Hi! I want to use from prefect.utilities.annotations.quote to speed up tasks that depend on large task outputs.
How can I use this pattern together with task.submit() ? See example code below:
Copy code
from prefect import task, flow
from prefect.utilities.annotations import quote
@task
def big_output():
return [1 for _ in range(10**7)]
@task
def big_input(x):
print(len(x))
@flow
def slow_flow():
x = big_output.submit()
big_input.submit(quote(x))
if __name__ == "__main__":
slow_flow()
I'm trying to get the big_input task to start quickly after big_output task, but I get this error:
Copy code
TypeError: object of type 'PrefectFuture' has no len()
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.