Hi everyone. I started using prefect a couple of months ago and it's been a good ride so far. By default, if the run() method succeeds for a task, Prefect sets the state to Success and records any returned data. I would however like to disable this recording of returned data because of how much memory it uses up. The task i run returns data of about 130Mb and I have thousands of raw data to map using this task. I couldn't find information about this online. How do you suggest i proceed? I should also mention that I'm running the flow without prefect server. Thank you for your time.

Hey @Oludayo, you should be able to stop this with

@task(checkpointing=False)

on the individual tasks that return the data.

Hello @Kevin Kho Thanks for the reply. I believe that is necessary for caching. I save the files in a dir locally so i can use the results later in another flow without having to run the task again.

Sorry I’m not understanding here. You need the data for another flow later, but you don’t want to checkpoint data because of the memory? Are you trying to turn off checkpointing, control the location of output yourself, and have the downstream Flow be responsible of fetching it at the location your specified?

Exactly. I would like to have the downstrem flow be responsible of fetching it at a specified location. I think I'll have to go with your recommendation of deleting the object and calling garbage collector. Thank you