Does anybody have a best practice for storing large data as a

Result

in 2.6+ ? I am thinking of passing data between tasks, persisted into remote storage as parquet, for example (e.g. using spark)

You can use any of our available storage blocks to store your results.

I am a bit confused since the default serializers imply that I can turn my data into bytes in memory. This is not possible for large data

Hey Evan, we don't serialize the result unless it is persisted because it has a lot of overhead and restricts the types of data users can pass between tasks.

I’m still a but fuzzy on the details so I’ll give an example. I want to pass a spark dataframe between tasks and save the intermediate out to s3. I should disable the in memory cache, and write a serializer that saves/loads a spark dataframe to s3?

Do you want the spark dataframe to pass between tasks in-memory or do you want the downstream task to read it from s3?

reading from s3 i think is fine

If you use a

cache_key_fn

and set

result_storage

to S3 it’ll persist the return value to S3 on completion. If there’s a cache hit, it’ll be pulled from S3. Otherwise, it’ll pass in memory.

that sounds like what I want I just want to control the persistence. I want to write partitioned parquet to s3

That'd be sweet! You could add it in a collection? Our template makes it pretty easy to get started.

what’s a collection?

https://docs.prefect.io/collections/overview/ — basically plugins.

tyty

Yes. https://docs.prefect.io/collections/catalog/ I’ll see if we can get a redirect up. Thanks! Welcome back!