https://prefect.io logo
j

Jeremy Knickerbocker

09/24/2020, 7:07 PM
Hi Everyone, hoping I am overlooking something simple. My drive filled up in my test environment, due to the results folder caching everything. I am running Prefect Core Server (v0.13.5) and I currently have checkpointing disabled. I read the docs here https://docs.prefect.io/core/advanced_tutorials/using-results.html and I believe I have everything configured properly. How can I a) make sure checkpointing is really disabled and b) change the global location of the results directory?
c

Chris White

09/24/2020, 7:13 PM
Hi Jeremy - very good question; the checkpointing flag that you are referencing is only toggleable for local / non-orchestrated runs (the value is hardcoded here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/cloud/task_runner.py#L215) for failure-recovery purposes. There are two things I recommend: - if you explicitly set
checkpoint=False
on all of your tasks, they will stop being checkpointed - you could open a feature request issue for making checkpointing toggleable when run against a backend, and we can look into it!
j

Jeremy Knickerbocker

09/24/2020, 7:22 PM
Thanks @Chris White for a quick reply, you saved me lots of time digging through code. Is it possible to change the default location of results? I think you surmised this, but I am using the LocalAgent so it is defaulting them all to ~/.prefect/results, which I believe comes from https://github.com/PrefectHQ/prefect/blob/dec10320dacf2ab7e90881feb09fb71c62729a69/src/prefect/engine/results/local_result.py#L42 so a few other potential options would be: • Subclass result or use LocalResult with the preferred directory (but per task) • Create a pull request to respect config.result_dir and then use config.home_dir + "/results" if result_dir does not exist (global) • Be super lazy and symlink results to where I want to it be (global but hacky) Any other thoughts on how I could avoid having to go edit a bunch of my existing flows
c

Chris White

09/24/2020, 7:25 PM
Yup you can configure that via the
dir
kwarg. For example:
Copy code
result = LocalResult(dir="~/my-own/dir")

with Flow("my-flow", result=result)...
Note that individual task results will always override the flow result configuration. If you really don’t want anything stored, you could also use the base
Result
class as your Flow’s result and I actually think that would do the trick for you!
j

Jeremy Knickerbocker

09/24/2020, 7:27 PM
Sorry, was editing my comment, you beat me to reply. Thank you, I think this will solve my immediate need, this was really helpful!
c

Chris White

09/24/2020, 7:27 PM
Ah I see now - glad to help!
@Marvin archive “How to disable checkpointing when running with Prefect Server?”