<@ULVA73B9P> I'd like to define a RemoteFileSystem...
# ask-marvin
j
@Marvin I'd like to define a RemoteFileSystem block. Can I supply my credentials to it using existing Secret blocks?
m
thought for 363.3 seconds
Short answer: Yes, but not by “pointing” a RemoteFileSystem at a Secret block directly. RemoteFileSystem has just two fields —
basepath
and
settings
— and
settings
is a plain dict passed to fsspec. You can load your Secret block at runtime and inject its values into
settings
. Common patterns: - Preferred (no secrets in the block): Leave
settings
empty and rely on your environment/identity (e.g., IAM role,
AWS_*
env vars, ADC for GCP). fsspec/s3fs/gcsfs will pick these up automatically. - Use a Secret block to populate
settings
when creating the RemoteFileSystem: Load the Secret, parse it (if it’s JSON), and pass the resulting dict into
settings
. Note: this will store the credentials inside the RemoteFileSystem block document. - Advanced: Create a tiny wrapper block that holds the Secret and builds a RemoteFileSystem on demand, so the RemoteFileSystem itself doesn’t persist secrets. Examples 1) Inject a Secret into RemoteFileSystem.settings (simple, but embeds creds in the block)
Copy code
from prefect.blocks.system import Secret
from prefect.filesystems import RemoteFileSystem
import json

# Secret contains JSON like: {"key": "...", "secret": "...", "region_name": "us-east-1"}
val = Secret.load("aws-creds").get()
settings = json.loads(val) if isinstance(val, str) else val  # parse if stored as JSON string

rfs = RemoteFileSystem(
    basepath="<s3://my-bucket/prefix/>",
    settings=settings
)
rfs.save("s3-storage", overwrite=True)
2) Wrapper block that references a Secret and builds the RemoteFileSystem at use-time (secrets stay in the Secret block)
Copy code
from typing import Any, Dict
from pydantic import Field
from prefect.blocks.core import Block
from prefect.blocks.system import Secret
from prefect.filesystems import RemoteFileSystem
import json

class RFSWithSecret(Block):
    basepath: str = Field(..., description="e.g., <s3://bucket/prefix%22|s3://bucket/prefix">)
    secret: Secret = Field(..., description="Secret containing JSON fsspec settings")

    def build(self) -> RemoteFileSystem:
        val = self.secret.get()
        settings: Dict[str, Any] = json.loads(val) if isinstance(val, str) else val
        return RemoteFileSystem(basepath=self.basepath, settings=settings)

# Usage
rfs_builder = RFSWithSecret(
    basepath="<s3://my-bucket/prefix>",
    secret=Secret.load("aws-creds")
)
rfs = rfs_builder.build()
Notes and best practices - For S3, typical fsspec keys are
key
,
secret
,
region_name
, and optionally
client_kwargs
like
endpoint_url
. For GCS, you might use
token
(path or dict), etc. The exact keys are protocol-specific. - If you save a RemoteFileSystem block after injecting credentials into
settings
, those credentials are stored in that block’s document (masked in the UI/CLI). If you want to avoid that, use environment/identity-based auth or the wrapper pattern above. - Secret.get() returns the underlying value. If you stored a JSON string, parse it with
json.loads
. Helpful links - RemoteFileSystem API: docs: RemoteFileSystem - Secret block API: docs: Secret block - Filesystems concept: docs: Filesystems - fsspec options (protocol-specific): fsspec docs Quick follow-up - Which protocol are you targeting (e.g., S3, GCS, Azure, SFTP/SMB)? I can suggest the exact
settings
keys for your backend. - Do you want to avoid embedding credentials in the RemoteFileSystem block, or is that acceptable for your setup?