In the short term, you can persist the results directly by writing out to a file using boto3 to your s3 bucket; in the long term, I’d expect you will be able to use your storage block and have it natively in prefect
a
Ahmed Ezzat
08/26/2022, 9:34 AM
Actually I don't really care about the data I'm trying to implement because I'm using prefect with dask which stores task data to the local disk and the disk itself temporarily as pods are getting destroyed regularly thus losing the task result when the engine asks for it