https://prefect.io logo
m

Matthew Blau

02/09/2021, 7:42 PM
Hello all, this should be a fairly simple question to answer, I just am having trouble locating the information in the documentation. I am wanting to enable checkpointing globally and I see that there is a way to do it temporarily via exporting the env variable, but I was hoping to know what I needed to add to the config.toml file for it to be a permanent feature. Thank you in advance!
c

Chris White

02/09/2021, 7:45 PM
Hi Matthew! Two options here: • when running against a Prefect API (Server or Cloud), checkpointing becomes the default with no additional config • if you want to enable it globally for local runs, add the following snippet to your user config.toml file:
Copy code
[flows]
checkpointing = true
m

Matthew Blau

02/09/2021, 7:49 PM
@Chris White ah, I see! So if it is already enabled then I just need to use results in order to utilize the checkpointing, yes? Where the results can, for example, store the info from an API call so that if a task down the chain fails we wouldn't need to re-call the API for that info, correct? If so, is there anything that I need to do other than setting the results location to take advantage of this?
c

Chris White

02/09/2021, 7:51 PM
Yup that is correct: checkpointing will begin working so long as your task has a result configured (which can also be configured at the flow level to apply to all tasks if you’d like) and your task actually returns data (i.e., it has a non-trivial
return
statement, which is what will be checkpointed).
m

Matthew Blau

02/09/2021, 8:10 PM
@Chris White awesome! Seems easy enough. I see in the documentation about using results it says "A result subclass that matches the storage backend of your
prefect.config.flows.storage
setting will automatically be applied to all tasks, if available; notably this is not yet supported for Docker Storage" is this to say that Flows ran in docker storage cannot take advantage of the checkpointing or is it saying that we can, just have to use a different storage method for the results?
c

Chris White

02/09/2021, 8:11 PM
The latter — there’s not a good sensible default when running a flow within Docker, so you just need to explicitly set a Result subclass, but otherwise it should work just as you expect!
m

Matthew Blau

02/09/2021, 8:37 PM
@Chris White ahh so I can just make my class a subclass of Task, configure the results, and the run function, and from there I should, in theory, be good to go? Seems easy enough
c

Chris White

02/10/2021, 12:02 AM
Yup exactly! Also note that the task decorator also accepts results so you don’t need to subclass if you don’t want to, although that is fully supported if you’d prefer it