Danny Vilela

    Danny Vilela

    4 months ago
    Is it possible to set a task to only write – not read – a result on Prefect Core? I was thinking of sub-classing a custom result class I wrote with flags
    enable_read
    and
    enable_write
    and just check those within the
    read
    and
    write
    methods, but maybe there’s something easier?
    Kevin Kho

    Kevin Kho

    4 months ago
    What is the use case here? I’m not getting why you would try to do this?
    Danny Vilela

    Danny Vilela

    4 months ago
    Let’s say you have a task that takes a few task outputs A, B, and C, runs some ETL on them, and writes them to a S3 location D. Only task D writes to S3 location D. Let’s assume tasks A, B, and C are “expensive” to compute, so you use Prefect’s checkpointing to write their results to S3. One day, you realize you made a big ol’ mistake and the logic in task D has some bug. You’d like to re-run the flow for the last week to overwrite results in S3 location D using the new and improved task D. Here, I’d want: • Tasks A, B, and C to use the cached results (they’re expensive computations). • Task D to not use the cached result, but write the new (fixed) result to S3. Does that make sense?
    Kevin Kho

    Kevin Kho

    4 months ago
    I think you should have caching on A,B,C and then you can invalidate the cache of D instead and that seems easier?
    Danny Vilela

    Danny Vilela

    4 months ago
    Ohh sure that does make more sense! Hahaha sorry for the dumb question @Kevin Kho you’re a much better engineer than I :thank-you: :thank-you: :thank-you:
    Kevin Kho

    Kevin Kho

    4 months ago
    No you’re too good, I won’t pull off that result subclass lol