Is it possible to set a task to only write – not r...
# prefect-community
d
Is it possible to set a task to only write – not read – a result on Prefect Core? I was thinking of sub-classing a custom result class I wrote with flags
enable_read
and
enable_write
and just check those within the
read
and
write
methods, but maybe there’s something easier?
k
What is the use case here? I’m not getting why you would try to do this?
d
Let’s say you have a task that takes a few task outputs A, B, and C, runs some ETL on them, and writes them to a S3 location D. Only task D writes to S3 location D. Let’s assume tasks A, B, and C are “expensive” to compute, so you use Prefect’s checkpointing to write their results to S3. One day, you realize you made a big ol’ mistake and the logic in task D has some bug. You’d like to re-run the flow for the last week to overwrite results in S3 location D using the new and improved task D. Here, I’d want: • Tasks A, B, and C to use the cached results (they’re expensive computations). • Task D to not use the cached result, but write the new (fixed) result to S3. Does that make sense?
k
I think you should have caching on A,B,C and then you can invalidate the cache of D instead and that seems easier?
d
Ohh sure that does make more sense! Hahaha sorry for the dumb question @Kevin Kho you’re a much better engineer than I 🙏 🙏 🙏
k
No you’re too good, I won’t pull off that result subclass lol