Ling Chen

    Ling Chen

    6 months ago
    Hi Community! Are there guidelines to free up the previous tasks' memory after they are finished? We are running memory intensive pipelines and run into out of memory error. The output of each task is only needed in the next task, but not in the subsequent tasks. Is there room for memory optimization?
    Kevin Kho

    Kevin Kho

    6 months ago
    You can persist the data and then just pass the location downstream to be loaded in. Are you doing that already?
    Ling Chen

    Ling Chen

    6 months ago
    When you say persist the data, are you referring to storing the data into disk and load it back in the next task?
    Kevin Kho

    Kevin Kho

    6 months ago
    yeah like
    <http://df.to|df.to>_parquet
    so that itโ€™s not held in memory because Prefect holds the mapped results and then you pass the location string instead of the df.
    Ling Chen

    Ling Chen

    6 months ago
    Cool. I can do that. But just to understand Prefect a bit better, there is no easy way to clear the memory of task outputs? So python gc.collect has no effect on task outputs?
    Anna Geller

    Anna Geller

    6 months ago
    Lastly, you can always simply delete Python objects if you no longer need them:
    del df
    Ling Chen

    Ling Chen

    6 months ago
    Cool cool. Something like del df and gc.collect() should work then. Thanks!
    Anna Geller

    Anna Geller

    6 months ago
    Looks like SO Python folks also recommend just that ๐Ÿ˜„
    del my_object
    gc.collect()
    https://stackoverflow.com/a/1316793/9509388
    Kevin Kho

    Kevin Kho

    6 months ago
    You can if you do it inside the task yep