https://prefect.io logo
Title
j

Josh Paulin

09/30/2022, 7:46 PM
Hello. Is there any best practice for handling large task results? I’ve got a task that can return >5GB of data as a result, and I assume that’s what’s causing the flow to slowdown creating and running subsequent tasks.
1
j

Jeff Hale

09/30/2022, 8:26 PM
Result handling is a high priority that @Zanie is working on now. You can follow the tracking PR here.
👀 1
z

Zanie

09/30/2022, 8:29 PM
Passing that much data around is always going to be expensive though. I’d consider persisting it to a location then returning the reference to that location and then loading it again when necessary.
We’ll have an option to automatically drop things from memory (as a part of the linked work) which will do something like that for you.
j

Josh Paulin

10/14/2022, 3:42 PM
Hello. Wanted to circle back on this since it looks like this was just released with 2.6.0 (🎉) Was the ability to drop from memory added along with this? Don’t see it in the docs anywhere, but I do see it still what looks like could be it still listed as planned in GitHub.
z

Zanie

10/17/2022, 1:40 PM
There's an open pull request to add that functionality, it'll probably be in the next release
👍 1
🙏 1