Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey everyone!

A few months ago, I posted a discussion on optimizing for high memory usage in the <#C03D12VV4NN|>: <https://prefect-community.slack.com/archives/C03D12VV4NN/p1671793942895089>

We're still running into this issue.
Idk if this apparent build up of memory usage is a bug with prefect or if it is something we can control

I'm just some guy (not associated with prefect org), but what TaskRunner are you using?

Hey <@U04BVPXAF6U>, I'm using the `SequentialTaskRunner`.

I've tried setting `cache_result_in_memory` as `False`, but that only moves the problem to the Disk Usage.
Flows now crash with `OSError: code 28 No space on device`

More context on the tasks:
• About 20-30 inputs at a time
• `task_a.map` -&gt; `task_a_states` -&gt; `task_a_results`
• `task_a_results` are used by `task_b` .`map`, to give `task_b_result`
• Finally, `task_c` takes in `task_b_results` and returns `None`.


I previously used the `ConcurrentTaskRunner` and found that the memory usage builds up even though the the input is already consumed by the end of `task_c`, and the new batch must begin.