Thread
#prefect-community
    Mike Wochner

    Mike Wochner

    1 year ago
    Is it possible to disable automatic input caching? The problem with the following flow setup is that the today_date variable seems to be chached from previous days, which is not what I intend. Is it possible to avoid this input caching or a better way to structure this. The reason the caputuring of the date happens at the beginning of the code, is that the entire flow can run for longer times and enter the next day.
    with Flow('Example') as flow:
    
        today_date = datetime.date.today().strftime("%Y-%m-%d")
    
        data = extract_data(security_list, today_date)
        load_data(data)
    
        ...
    
        
        more_data = extract_more_data(security_list)
        load_more_data(more_data, today_date)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Mike Wochner, we override checkpointing on the cloud/server side so you need to disable it on the task level like
    @task(checkpointing=False)
    . If you are concerned about memory, some users provide the file path for the output files so it overwrites each time the flow is run, and you’ll be able to restart the last flow in case it fails
    Mike Wochner

    Mike Wochner

    1 year ago
    Thanks for the answer. I have checkpointing disabled for individual tasks, though the saving of todays date happens outside of a task. Would it be best practice to make a task from it?
    And then disable checkpointing for that specific task?
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh my bad. Sorry, I didn’t read the full question. The date there will be serialized when the flow is registered, unless you use script-based storage. The fix here is to use
    prefect.context.get("today")
    , or you can have a task that returns the current date. The task execution will be deferred, but the content of the Flow is not, which is why we run into this.
    Mike Wochner

    Mike Wochner

    1 year ago
    Makes sense. Thanks for the help on this!
    Kevin Kho

    Kevin Kho

    1 year ago
    Script based storage would fix this too I think. TLDR is you can just do
    LocalStorage("location of file", stored_as_script=True)
    Mike Wochner

    Mike Wochner

    1 year ago
    Thanks, Kevin. Both are new concepts, but both helpful. Appreciate your help.