Hey, We are developing a flow where a lot of JSON ...
# prefect-community
l
Hey, We are developing a flow where a lot of JSON records get queried (in a Task). To save on memory resources, I've configured Result Persistence on the Task, but I'm now noticing a build up of memory usage of a serializer object. When are results serialized and when is this memory released?
Build up of memory usage by serializer.py:
Copy code
/usr/local/lib/python3.9/site-packages/prefect/serializers.py:147: size=70.3 MiB, count=427649, average=172 B
...
/usr/local/lib/python3.9/site-packages/prefect/serializers.py:147: size=73.4 MiB, count=450775, average=171 B
Task decorator:
Copy code
@task(cache_result_in_memory=False)
k
Hey @Lennert Van de Velde how did you configure the result persistence on the task? What is the output of your
PREFECT_RESULTS_PERSIST_BY_DEFAULT
when you run
prefect config view
?
l
Hi Kalise, I've already put up an issue on Github, so it is close to being solved I believe 🙂 https://github.com/PrefectHQ/prefect/issues/7624