Sunny Shah
11/10/2025, 6:45 AMfrom prefect.cache_policies import NO_CACHE
@task(cache_policy=NO_CACHE)
def some_task():
do_something
Is there a way to globalize the cache_policy? (edited)Marvin
11/10/2025, 6:45 AMMarvin
11/10/2025, 6:53 AMcache_policy=NO_CACHE on every task.
Options:
- Per-profile / environment (recommended)
- Default all tasks to no-cache unless explicitly overridden:
prefect config set PREFECT_TASKS_DEFAULT_NO_CACHE=true
or set the env var:
export PREFECT_TASKS_DEFAULT_NO_CACHE=true
- Hard-disable caching everywhere (ignore any task’s cache policy):
prefect config set PREFECT_TASKS_DISABLE_CACHING=true
or
export PREFECT_TASKS_DISABLE_CACHING=true
- Code-level convenience (project-local)
- Create a wrapper so you don’t repeat the argument:
from functools import partial
from prefect import task
from prefect.cache_policies import NO_CACHE
nocache_task = partial(task, cache_policy=NO_CACHE)
@nocache_task
def some_task():
do_something
Notes:
- PREFECT_TASKS_DEFAULT_NO_CACHE=true sets the default to NO_CACHE for tasks that don’t specify a policy; any task that explicitly sets a cache_policy will still use that.
- PREFECT_TASKS_DISABLE_CACHING=true is stronger: it turns off all cache reads/writes regardless of any task’s settings.
If you can share whether you want this applied globally across your whole environment or just within a single project/run, I can tailor the setup steps further.Sunny Shah
11/10/2025, 7:11 AMMarvin
11/10/2025, 7:11 AMMarvin
11/10/2025, 7:12 AMprefect config set PREFECT_TASKS_DEFAULT_NO_CACHE=true
- Or set via environment variable (for a single shell/session):
export PREFECT_TASKS_DEFAULT_NO_CACHE=true # bash/zsh
# PowerShell: $env:PREFECT_TASKS_DEFAULT_NO_CACHE = 'true'
- For just one run:
PREFECT_TASKS_DEFAULT_NO_CACHE=true python my_flow.py
- Hard-disable caching everywhere (ignore any task’s cache policy)
- Persist in the active Prefect profile:
prefect config set PREFECT_TASKS_DISABLE_CACHING=true
- Or set via environment variable:
export PREFECT_TASKS_DISABLE_CACHING=true # bash/zsh
# PowerShell: $env:PREFECT_TASKS_DISABLE_CACHING = 'true'
- For just one run:
PREFECT_TASKS_DISABLE_CACHING=true python my_flow.py
2) Project-local code wrapper (don’t repeat cache_policy)
- Create a local decorator you can reuse:
from functools import partial
from prefect import task, flow
from prefect.cache_policies import NO_CACHE, DEFAULT
# Project-local decorator that sets NO_CACHE by default
nocache_task = partial(task, cache_policy=NO_CACHE)
@nocache_task
def compute_a(x: int) -> int:
return x + 1
# You can still override per-task when needed
@task(cache_policy=DEFAULT)
def compute_b(x: int) -> int:
return x * 2
@flow
def my_flow():
a = compute_a(1)
b = compute_b(a)
return b
if __name__ == "__main__":
my_flow()
Notes:
- PREFECT_TASKS_DEFAULT_NO_CACHE=true sets the default to NO_CACHE for tasks that don’t specify a cache_policy.
- PREFECT_TASKS_DISABLE_CACHING=true is stronger and turns off all caching reads/writes regardless of individual task settings.
If you want this applied only in CI or only for a specific deployment, I can share examples tailored to those environments too.Sunny Shah
11/10/2025, 7:56 AMMarvin
11/10/2025, 7:56 AMMarvin
11/10/2025, 7:59 AMname: Deploy Prefect flows
on:
push:
branches: [ main ]
jobs:
deploy:
runs-on: ubuntu-latest
env:
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_TASKS_DEFAULT_NO_CACHE: "true" # key line
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt
- run: prefect config validate
- run: prefect deploy ./flows/my_flow.py:my_flow -n my-deployment
GitLab CI (job-level env)
deploy:
image: python:3.11
variables:
PREFECT_API_URL: $PREFECT_API_URL
PREFECT_API_KEY: $PREFECT_API_KEY
PREFECT_TASKS_DEFAULT_NO_CACHE: "true" # key line
script:
- pip install -r requirements.txt
- prefect config validate
- prefect deploy ./flows/my_flow.py:my_flow -n my-deployment
Dockerfile for your runtime image (ensures flows always run with NO_CACHE)
FROM python:3.11-slim
ENV PREFECT_TASKS_DEFAULT_NO_CACHE=true
# ...install deps, copy code, etc.
Kubernetes worker (Helm values or Pod env for your worker container)
# values.yaml snippet for a worker
env:
- name: PREFECT_TASKS_DEFAULT_NO_CACHE
value: "true"
B) Hard-disable caching everywhere (ignore any task’s cache policy)
GitHub Actions
env:
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
PREFECT_TASKS_DISABLE_CACHING: "true" # key line
GitLab CI
variables:
PREFECT_API_URL: $PREFECT_API_URL
PREFECT_API_KEY: $PREFECT_API_KEY
PREFECT_TASKS_DISABLE_CACHING: "true" # key line
Dockerfile
ENV PREFECT_TASKS_DISABLE_CACHING=true
Kubernetes worker
env:
- name: PREFECT_TASKS_DISABLE_CACHING
value: "true"
Optional: persist in the CI runner’s Prefect profile (ephemeral, but useful for the CI job itself)
prefect config set PREFECT_TASKS_DEFAULT_NO_CACHE=true
# or
prefect config set PREFECT_TASKS_DISABLE_CACHING=true
Quick guidance
- If your flows are executed by a long-running worker (process, Docker, or Kubernetes): set the env var where the worker runs (Dockerfile, Helm chart, systemd env).
- If you run flows directly in CI (e.g., ad hoc python my_flow.py or prefect run deployment within the pipeline): set the env var in the CI job.
- Choose DEFAULT_NO_CACHE if you still want the option for specific tasks to opt-in to caching. Choose DISABLE_CACHING to turn caching off completely.