Jackson Nguyen
12/02/2025, 9:43 PMMarvin
12/02/2025, 9:43 PMMarvin
12/02/2025, 9:49 PMbasepath and settings — and settings is a plain dict passed to fsspec. You can load your Secret block at runtime and inject its values into settings.
Common patterns:
- Preferred (no secrets in the block): Leave settings empty and rely on your environment/identity (e.g., IAM role, AWS_* env vars, ADC for GCP). fsspec/s3fs/gcsfs will pick these up automatically.
- Use a Secret block to populate settings when creating the RemoteFileSystem: Load the Secret, parse it (if it’s JSON), and pass the resulting dict into settings. Note: this will store the credentials inside the RemoteFileSystem block document.
- Advanced: Create a tiny wrapper block that holds the Secret and builds a RemoteFileSystem on demand, so the RemoteFileSystem itself doesn’t persist secrets.
Examples
1) Inject a Secret into RemoteFileSystem.settings (simple, but embeds creds in the block)
from prefect.blocks.system import Secret
from prefect.filesystems import RemoteFileSystem
import json
# Secret contains JSON like: {"key": "...", "secret": "...", "region_name": "us-east-1"}
val = Secret.load("aws-creds").get()
settings = json.loads(val) if isinstance(val, str) else val # parse if stored as JSON string
rfs = RemoteFileSystem(
basepath="<s3://my-bucket/prefix/>",
settings=settings
)
rfs.save("s3-storage", overwrite=True)
2) Wrapper block that references a Secret and builds the RemoteFileSystem at use-time (secrets stay in the Secret block)
from typing import Any, Dict
from pydantic import Field
from prefect.blocks.core import Block
from prefect.blocks.system import Secret
from prefect.filesystems import RemoteFileSystem
import json
class RFSWithSecret(Block):
basepath: str = Field(..., description="e.g., <s3://bucket/prefix%22|s3://bucket/prefix">)
secret: Secret = Field(..., description="Secret containing JSON fsspec settings")
def build(self) -> RemoteFileSystem:
val = self.secret.get()
settings: Dict[str, Any] = json.loads(val) if isinstance(val, str) else val
return RemoteFileSystem(basepath=self.basepath, settings=settings)
# Usage
rfs_builder = RFSWithSecret(
basepath="<s3://my-bucket/prefix>",
secret=Secret.load("aws-creds")
)
rfs = rfs_builder.build()
Notes and best practices
- For S3, typical fsspec keys are key, secret, region_name, and optionally client_kwargs like endpoint_url. For GCS, you might use token (path or dict), etc. The exact keys are protocol-specific.
- If you save a RemoteFileSystem block after injecting credentials into settings, those credentials are stored in that block’s document (masked in the UI/CLI). If you want to avoid that, use environment/identity-based auth or the wrapper pattern above.
- Secret.get() returns the underlying value. If you stored a JSON string, parse it with json.loads.
Helpful links
- RemoteFileSystem API: docs: RemoteFileSystem
- Secret block API: docs: Secret block
- Filesystems concept: docs: Filesystems
- fsspec options (protocol-specific): fsspec docs
Quick follow-up
- Which protocol are you targeting (e.g., S3, GCS, Azure, SFTP/SMB)? I can suggest the exact settings keys for your backend.
- Do you want to avoid embedding credentials in the RemoteFileSystem block, or is that acceptable for your setup?