<@ULVA73B9P> In Prefect 3, if I set these env vari...
# ask-marvin
n
@Marvin In Prefect 3, if I set these env variables, PREFECT_RESULTS_PERSIST_BY_DEFAULT=true and PREFECT_DEFAULT_RESULT_STORAGE_BLOCK=s3-bucket/my-s3-block, are ALL task results persisted by default on S3? Do I need to set other env variables to achieve this?
m
thought for 274.5 seconds
Short answer: Yes. In Prefect 3.x, setting -
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
, and -
PREFECT_DEFAULT_RESULT_STORAGE_BLOCK=s3-bucket/my-s3-block
(or the canonical
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK=s3-bucket/my-s3-block
) will cause flow and task results to be persisted by default to that S3 block, unless you explicitly override persistence on a given flow/task. Details and gotchas: - Either env var name for the storage block works:
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK
(recommended) or
PREFECT_DEFAULT_RESULT_STORAGE_BLOCK
(also supported). - Precedence rules: - Task/flow-level
persist_result
or
result_storage
parameters override the env vars. - If a task has
@task(persist_result=False)
, it will not persist even with the env vars set. - The S3 block must exist in your workspace and be loadable from the run environment. - Make sure the runtime has: - The Prefect AWS collection installed (
prefect-aws
) so the S3 block can write to S3. - IAM permissions to write to the target bucket (e.g., via the CodeBuild role or AWS_* env vars). - Optional (not required): If you want non-pickle serialization, set
PREFECT_RESULTS_DEFAULT_SERIALIZER=json
. No other Prefect env vars are strictly required for default result persistence to S3 beyond the two you listed.
n
@Marvin So what does
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
do? What is the difference?
m
thought for 88.7 seconds
Great question. Think of two layers of defaults: - PREFECT_RESULTS_PERSIST_BY_DEFAULT (global default for flows and tasks) - PREFECT_TASKS_DEFAULT_PERSIST_RESULT (task-only default that overrides the global default) What PREFECT_TASKS_DEFAULT_PERSIST_RESULT does: - It sets a workspace-wide default for all tasks’ persistence behavior. - It only applies when a task doesn’t have an explicit
persist_result
set. - It takes precedence over both the flow’s
persist_result
and
PREFECT_RESULTS_PERSIST_BY_DEFAULT
. Precedence for a task’s persistence (highest to lowest): 1) Task-level
persist_result
on the decorator 2) PREFECT_TASKS_DEFAULT_PERSIST_RESULT 3) Flow-level
persist_result
4) PREFECT_RESULTS_PERSIST_BY_DEFAULT 5) Default: False (no persistence) Concrete examples: - If you set
PREFECT_TASKS_DEFAULT_PERSIST_RESULT=false
and your flow has
@flow(persist_result=True)
, tasks will NOT persist (the task default overrides the flow). - If you set
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
but do not set
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
, tasks will persist unless the flow/task says otherwise. - If a task has
@task(persist_result=True)
, it will persist regardless of either env var. Env var name: - The correct env var is
PREFECT_TASKS_DEFAULT_PERSIST_RESULT
(no aliases).