hey guys, hopefully this is an easy / noob level resolution that's required: I want to get the count of records returned in a dataframe from a task to pass through to another task like so:

with Flow('test_flow',executor=LocalExecutor()) as flow:

df = sometask_that_returns_a_df()

count_of_records_in_df = len(df.index)

execute_task_2(count_of_records_in_df)

Obvs compilation fails because the return type of sometask is a functiontask

Hi @Adam Everington, you would have to wrap it into a task, then you can use it within the

with Flow()

constructor:

@task
def get_len_of_df(df):
    return len(df.index) 

with Flow('test_flow',executor=LocalExecutor()) as flow:
    df = sometask_that_returns_a_df()
    count_of_records_in_df = get_len_of_df(df)
    execute_task_2(count_of_records_in_df)

awesome, thanks @Anna Geller!

It depends on many factors - what type of storage do you use, whether you use Dask. What if your class DTO would also be a Prefect task? Here is a docs page showing how to subclass the Task class

class DTO(Task):
    def run(self, data, data_count):
        # do sth
        return data

You can access DTO as long as it is serializeable with

cloudpickle

, which I think it should be.