Is it possible to disable automatic input caching? The problem with the following flow setup is that the today_date variable seems to be chached from previous days, which is not what I intend. Is it possible to avoid this input caching or a better way to structure this. The reason the caputuring of the date happens at the beginning of the code, is that the entire flow can run for longer times and enter the next day.

with Flow('Example') as flow:

    today_date = datetime.date.today().strftime("%Y-%m-%d")

    data = extract_data(security_list, today_date)
    load_data(data)

    ...

    
    more_data = extract_more_data(security_list)
    load_more_data(more_data, today_date)

Hi @Mike Wochner, we override checkpointing on the cloud/server side so you need to disable it on the task level like

@task(checkpointing=False)

. If you are concerned about memory, some users provide the file path for the output files so it overwrites each time the flow is run, and you’ll be able to restart the last flow in case it fails

Thanks for the answer. I have checkpointing disabled for individual tasks, though the saving of todays date happens outside of a task. Would it be best practice to make a task from it?

Oh my bad. Sorry, I didn’t read the full question. The date there will be serialized when the flow is registered, unless you use script-based storage. The fix here is to use

prefect.context.get("today")

, or you can have a task that returns the current date. The task execution will be deferred, but the content of the Flow is not, which is why we run into this.

Makes sense. Thanks for the help on this!

Script based storage would fix this too I think. TLDR is you can just do

LocalStorage("location of file", stored_as_script=True)

Thanks, Kevin. Both are new concepts, but both helpful. Appreciate your help.