Hello guys,
I have a question related to the task RAM usage - it is possible to save results using S3 storage or other storage without keeping all results in RAM?
This is needed because I would like to process a large file, that is split into x subfiles, load each subfile in a different task, perform necessary operations, save it in the pickle file and then load that pickle file in the other task.
The goal is to keep organised tasks in the prefect flow(I would like to track each subfile task operation in the prefect tasks) and keep as minimal RAM usage as possible by not storing all data in the RAM(need to store just 1 subfile at the time in the RAM).
For now I'm trying to use S3 storage results(
https://docs.prefect.io/orchestration/execution/storage_options.html#aws-s3), but it seems it is not free up RAM memory when result is saved into the pickle file.
Any ideas related with this problem?