Hey,
We are developing a flow where a lot of JSON records get queried (in a Task). To save on memory resources, I've configured Result Persistence on the Task, but I'm now noticing a build up of memory usage of a serializer object. When are results serialized and when is this memory released?
Lennert Van de Velde
11/21/2022, 3:06 PM
Build up of memory usage by serializer.py:
Copy code
/usr/local/lib/python3.9/site-packages/prefect/serializers.py:147: size=70.3 MiB, count=427649, average=172 B
...
/usr/local/lib/python3.9/site-packages/prefect/serializers.py:147: size=73.4 MiB, count=450775, average=171 B
Task decorator:
Copy code
@task(cache_result_in_memory=False)
k
Kalise Richmond
11/22/2022, 5:49 PM
Hey @Lennert Van de Velde how did you configure the result persistence on the task? What is the output of your
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.