Hey, I have a worker in ECS and a few Prefect deployments. I want to enable cache for Prefect Tasks. How does that work in ECS Tasks spawned by the worker?
brodul
06/15/2023, 3:18 PM
If i understand correctly all the cache is in
PREFECT_LOCAL_STORAGE_PATH
and that is currently the only supported option.
j
Jeff Hale
06/15/2023, 5:16 PM
You can cache the results to external storage such as an S3 bucket.
b
brodul
06/16/2023, 7:25 AM
Is it enough to use the Prefect Result S3 ?
j
Jeff Hale
06/16/2023, 12:54 PM
If you have an S3Bucket block from prefect_aws you can do something like this:
Copy code
from prefect.filesystems import S3
s3_block = S3.load("myawsblock")
@flow(persist_result=True, result_storage=s3_block)
def my_func():
# call tasks the cache results
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.