Hey everyone! A few months ago, I posted a discussion on optimizing for high memory usage in the <#...

Kelvin DeCosta

02/14/2023, 7:10 AM

Hey everyone! A few months ago, I posted a discussion on optimizing for high memory usage in the #C03D12VV4NN: https://prefect-community.slack.com/archives/C03D12VV4NN/p1671793942895089 We're still running into this issue. Idk if this apparent build up of memory usage is a bug with prefect or if it is something we can control

Joshua Grant

02/21/2023, 9:03 PM

I'm just some guy (not associated with prefect org), but what TaskRunner are you using?

Kelvin DeCosta

03/15/2023, 9:20 AM

Hey @Joshua Grant, I'm using the

SequentialTaskRunner

. I've tried setting

cache_result_in_memory

False

, but that only moves the problem to the Disk Usage. Flows now crash with

OSError: code 28 No space on device

Kelvin DeCosta

03/15/2023, 9:24 AM

More context on the tasks: • About 20-30 inputs at a time •

task_a.map

task_a_states

task_a_results

•

task_a_results

are used by

task_b

map

, to give

task_b_result

• Finally,

task_c

takes in

task_b_results

and returns

None

Kelvin DeCosta

03/15/2023, 9:25 AM

I previously used the

ConcurrentTaskRunner

and found that the memory usage builds up even though the the input is already consumed by the end of

task_c

, and the new batch must begin.

6 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.