Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello. Is there any best practice for handling large task results? I’ve got a task that can return &gt;5GB of data as a result, and I assume that’s what’s causing the flow to slowdown creating and running subsequent tasks.

Result handling is a high priority that <@U01CEUST9B5> is working on now. You can follow the tracking PR <https://github.com/PrefectHQ/prefect/pull/6908|here>.

Passing that much data around is always going to be expensive though. I’d consider persisting it to a location then returning the reference to that location and then loading it again when necessary.

We’ll have an option to automatically drop things from memory (as a part of the linked work) which will do something like that for you.

Hello. Wanted to circle back on this since it looks like this was just released with 2.6.0 (:tada:)

Was the ability to drop from memory added along with this? Don’t see it in the docs anywhere, but I do see it still what looks like could be it still listed as planned in GitHub.

There's an open pull request to add that functionality, it'll probably be in the next release