Evan Curtin
10/19/2022, 2:57 PMResult
in 2.6+ ? I am thinking of passing data between tasks, persisted into remote storage as parquet, for example (e.g. using spark)Jean Luciano
10/19/2022, 3:46 PMEvan Curtin
10/19/2022, 8:55 PMZanie
10/20/2022, 1:39 PMEvan Curtin
10/20/2022, 1:44 PMZanie
10/20/2022, 2:23 PMEvan Curtin
10/20/2022, 2:25 PMZanie
10/20/2022, 2:31 PMcache_key_fn
and set result_storage
to S3 it’ll persist the return value to S3 on completion. If there’s a cache hit, it’ll be pulled from S3. Otherwise, it’ll pass in memory.cache_result_in_memory=False
Evan Curtin
10/20/2022, 2:34 PMZanie
10/21/2022, 2:32 PMEvan Curtin
10/21/2022, 2:33 PMZanie
10/21/2022, 2:57 PMEvan Curtin
10/21/2022, 2:58 PMJeff Hale
11/15/2022, 1:27 PM