https://prefect.io logo
Title
a

Ahmed Ezzat

08/24/2022, 3:59 AM
I can't find any references in 2.0 docs mentioning
result_filesystem
it's responsible for handling task result storage. I'm trying to store task result on s3 bucket
1
c

Christopher Boyd

08/24/2022, 1:14 PM
Hi Ahmed, I think currently task results are local only, and there is a a feature request being developed to persist on remote filesystems. https://docs.prefect.io/concepts/tasks/?h=result#task-results
In the short term, you can persist the results directly by writing out to a file using boto3 to your s3 bucket; in the long term, I’d expect you will be able to use your storage block and have it natively in prefect
a

Ahmed Ezzat

08/26/2022, 9:34 AM
Actually I don't really care about the data I'm trying to implement because I'm using prefect with dask which stores task data to the local disk and the disk itself temporarily as pods are getting destroyed regularly thus losing the task result when the engine asks for it