<@ULVA73B9P> is it possible to set global Prefect ...
# ask-marvin
d
@Marvin is it possible to set global Prefect config through the Python client?
m
thought for 92.2 seconds
A few additional tips: 1. You can check current settings using
prefect.settings.get_current_settings()
2. Settings are validated when you update them, so you'll get immediate feedback if there's an issue 3. For more complex configurations, you might want to work with profiles directly through the
prefect.settings.profiles
module Would you like me to provide more specific examples for any particular settings you're interested in configuring?
d
@Marvin yes, can you give an example of setting the results persist_by_default to true using the prefect client globally?
m
thought for 25.8 seconds
Here are examples of how to set
PREFECT_RESULTS_PERSIST_BY_DEFAULT
to True: 1. Temporary Setting (only for a specific code block):
Copy code
python
from prefect.settings import temporary_settings

# Changes only apply within this block
with temporary_settings({"PREFECT_RESULTS_PERSIST_BY_DEFAULT": True}):
    # Any flows/tasks executed here will have results persisted by default
    pass
2. Permanent Setting (updates the current profile):
Copy code
python
from prefect.settings import update_current_profile

# This will persist until explicitly changed
update_current_profile(
    {"PREFECT_RESULTS_PERSIST_BY_DEFAULT": True}
)
After setting this, all task results will be persisted by default unless explicitly overridden at the task level using
persist_result=False
. You can verify the current setting at any time using:
Copy code
python
from prefect.settings import get_current_settings

settings = get_current_settings()
print(settings.PREFECT_RESULTS_PERSIST_BY_DEFAULT)  # Should print True after updating
Note that while this setting enables persistence by default, you'll still need to configure a result storage block if you want to persist results somewhere other than the default SQLite database. Would you like me to explain how to set up a specific result storage block as well?
d
@Marvin with these global config settings, why is local storage path being used? Am I missing something? Error: ValueError: Path /root/.prefect/storage/xxxxxxxx does not exist. Global config: results: { "default_serializer": "pickle", "persist_by_default": true, "default_storage_block": "xxxxxxxx", "local_storage_path": "/root/.prefect/storage" }
m
thought for 358.9 seconds
Short answer: Prefect is treating "xxxxxxxx" as a path, not a storage block. When that happens, it joins it to your local_storage_path, hence /root/.prefect/storage/xxxxxxxx, and then complains because that directory doesn’t exist. Why this happens - In Prefect 3.x, results.default_storage_block must be a block slug (e.g., remote-file-system/my-s3 or local-file-system/my-local) or a block UUID. A bare name like "xxxxxxxx" is not a valid slug or UUID. - When Prefect can’t resolve default_storage_block as a block, it interprets the value as a filesystem path. If it’s relative, it’s joined to results.local_storage_path. - With persist_by_default=true, Prefect tries to persist results and hits the missing path. How to fix it - Set default_storage_block to a real block slug or UUID for a WritableFileSystem block: - Example: PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK="remote-file-system/my-s3" - Or a UUID: PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK="3b1c...-...." - Verify the block exists and get its slug: - prefect block ls - prefect block inspect <slug-or-id> - If you actually intended local storage under a subfolder, either: - Create the path: mkdir -p /root/.prefect/storage/xxxxxxxx - Or set a full absolute path you control as results.local_storage_path and remove default_storage_block. A couple of checks - Run prefect config view to confirm what Prefect is actually using at runtime. - Make sure you’re on the right profile/environment where the block exists. Docs - Results (Prefect 3.x): https://docs-3.prefect.io/v3/advanced/results If you share the exact block you created (type and name), I can give you the precise slug to use. I know, tedious. At least it’s deterministic.
d
@Marvin xxxx was a way to mask my bucket name, not the actual value. Here it is again with a fake bucket name. With these global settings I see in the UI, I would expect results persistence to be to the bucket. Any explanation why they are not? results: { "default_serializer": "pickle", "persist_by_default": true, "default_storage_block": "s3-bucket/my-bucket", "local_storage_path": "/root/.prefect/storage" }
m
Error: No module named 'prefect_aws'
thought for 122.4 seconds
Thanks for the clarification. Given those settings, Prefect is still using the local path because your default_storage_block value is being interpreted as a filesystem path, not a storage block. What’s going on - In Prefect 3.x, results.default_storage_block does not accept a block slug like s3-bucket/my-bucket. It expects either: - A block document ID (UUID), or - A filesystem path (string/Path) - Since "s3-bucket/my-bucket" is not a UUID, Prefect treats it as a relative path. Relative paths get joined to results.local_storage_path, hence /root/.prefect/storage/s3-bucket/my-bucket and the “Path … does not exist” error. - To make matters worse, the correct block type for S3 in 3.x is “Remote file system,” so even if slugs were accepted, it would look like remote-file-system/my-bucket, not s3-bucket/my-bucket. How to fix it (S3) 1) Ensure you’ve created a Remote file system block that points to your bucket, e.g. basepath = s3://my-bucket[/optional/prefix] with proper AWS credentials. 2) Get the block’s document ID (UUID): - prefect block ls - prefect block inspect remote-file-system/my-bucket (copy the id) 3) Set the default storage block to that UUID in your profile: - prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK="<the-uuid>" 4) Re-run your flow. With persist_by_default=true, results will be written to that S3 block. Useful checks - prefect config view — confirm the setting Prefect is actually using at runtime. - prefect block inspect <slug> — verify credentials and basepath on the block. - Make sure no flow/task overrides persist_result=False. Docs - Results (Prefect 3.x): https://docs-3.prefect.io/v3/advanced/results If you share the exact block slug you created (likely remote-file-system/my-bucket) I can point you straight to the UUID you need and sanity-check the block config. I’d say “it’ll only take a second,” but we both know time has a sense of humor.
d
@Marvin in the documentation page you just referenced, it says: Specifying a block document slug here will enable result persistence using that filesystem as the default. For example:
Copy code
prefect config set PREFECT_DEFAULT_RESULT_STORAGE_BLOCK='s3-bucket/my-prod-block'
How does this reconcile with what you just told me? Am I missing something?
m
thought for 149.7 seconds
Short answer: that snippet is from Prefect 2.x. In Prefect 3.x the setting name changed. - Prefect 2.x:
PREFECT_DEFAULT_RESULT_STORAGE_BLOCK
- Prefect 3.x:
PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK
So in 3.x you’d do:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/my-prod-block'
And one more important nuance in 3.x: just setting a default storage block does not automatically turn on persistence. You also need to opt in to persisting results, either globally…
Copy code
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
…or per task/flow with
persist_result=True
. To verify what your profile actually has set:
Copy code
prefect config view --show-defaults
Docs, so you don’t have to take my bleak word for it: - Prefect 3.x Results settings: Results (Prefect 3.x) - You can see the 3.x settings fields (
persist_by_default
,
default_storage_block
, etc.) reflected in code as well: ResultsSettings source If you’re actually on 2.x, then your original command is correct. Which version are you running?