Thread
#prefect-community
    Noah Holm

    Noah Holm

    1 year ago
    I’m using S3 storage for flows running with an ECS Agent. By default the flow’s tasks uses S3Result as outlined here. Is there any way to disable the task results for individual or all tasks in the flow while keeping the S3Storage on the flow?
    Amanda Wee

    Amanda Wee

    1 year ago
    It sounds like you might want to disable checkpointing, e.g., by setting
    PREFECT__FLOWS__CHECKPOINTING
    env var to false. I haven't tried this myself, but it's what I figured out from:https://docs.prefect.io/core/concepts/results.html#pipeline-persisted-results
    Noah Holm

    Noah Holm

    1 year ago
    Thanks Amanda! Reading through that page makes me realise I need to rethink my ideas a little here instead of actually disabling the checkpointing. For example, I might actually want to keep the S3Results and instead not pass actual results between tasks but a pointer to the results like uploading a big file to S3 within a task and pass the key for it to downstream tasks.
    Dylan

    Dylan

    1 year ago
    Hi @Noah Holm! When dealing with large volumes of data, I adopt the strategy you’ve outlined here 💯 . As an FYI, I believe you can remove a result from an individual task in a Flow by specifying
    result=None
    in your Task decorator.
    Noah Holm

    Noah Holm

    1 year ago
    Nice Dylan 🙂 I did try
    @task(result=None)
    but it didn’t make any difference. I guess it shouldn’t since that’s the default value so how would it know that I explicitly set it to None and that it wasn’t the default when the flow is evaluated.
    Dylan

    Dylan

    1 year ago
    Ahh true
    Hmm
    OH my mistake
    It’s
    checkpoint=False
    Noah Holm

    Noah Holm

    1 year ago
    Ah awesome! That kind of flag is exactly what I was looking when asking. Ty 🙏