Kevin Systrom
04/29/2020, 12:52 AMChris White
04/29/2020, 12:54 AMprefect.context
. I believe that your call to
with prefect.context():
is effectively clearing the cache between runsKevin Systrom
04/29/2020, 1:39 AMChris White
04/29/2020, 1:48 AMprefect.context['caches']
, so if you clear your context the cache will disappear; if you removed the with prefect.context():
block then you shouldn’t need to worry about the cache disappearing unexpectedly
- for the backend, sounds like you figured it out but to provide some more color: Prefect Core alone maintains state only as long as the Python process running your Flow. We have various options for how to run flows in a more stateful way (which includes how task caching is handled!), one of which is Prefect Server (our open source offering) — if you don’t want to deal with setting up the database, services, etc. Prefect Cloud is essentially a managed version of server (with more bells and whistles, e.g., authentication / user management / etc.). Note also that Cloud is designed to keep your code and data private (see, e..g, https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296)
For the best caching experience, I do recommend using one of our backendsKevin Systrom
04/29/2020, 1:50 AMChris White
04/29/2020, 1:51 AMKevin Systrom
04/29/2020, 1:51 AMChris White
04/29/2020, 1:52 AMKevin Systrom
04/29/2020, 1:56 AMChris White
04/29/2020, 1:57 AM