Hey team, I am trying to add external storage to t...
# ask-community
b
Hey team, I am trying to add external storage to to store the results of my tasks and flows. Although it seems to be working locally, when I try to deploy a flow using
prefect deploy
, I get this error:
Copy code
prefect.exceptions.ScriptError: Script at 'src/providers/Tapology/flows/data_ingestion.py' encountered an exception: TypeError('Result storage configuration must be persisted server-side. Please call `.save()` on your block before passing it in.')
An exception occurred.
The thing is that I have created the storage block directly in the Prefect web UI, and I'm loading it in my code, something like this:
Copy code
from prefect import flow
from prefect.filesystems import RemoteFileSystem
from prefect.serializers import JSONSerializer
from prefect.futures import as_completed
from prefect.task_runners import ThreadPoolTaskRunner

# Load the storage block you saved previously
r2_storage = RemoteFileSystem.load("r2-persistent-storage")

# Choose serializer for your payloads
json_ser = JSONSerializer()

@flow(
    log_prints=True,
    task_runner=ThreadPoolTaskRunner(max_workers=3),
    persist_result=True,
    result_storage=r2_storage,
    result_serializer=json_ser,
)
def my_flow_function():
    [...]
Another thing to note is that this code is working for already existing deployments, the error is only thrown when trying to deploy a new deployment. Any idea whats going on?
@Marvin I am trying to add external storage to to store the results of my tasks and flows. Although it seems to be working locally, when I try to deploy a flow using
prefect deploy
, I get this error:
Copy code
prefect.exceptions.ScriptError: Script at 'src/providers/Tapology/flows/data_ingestion.py' encountered an exception: TypeError('Result storage configuration must be persisted server-side. Please call `.save()` on your block before passing it in.')
An exception occurred.
The thing is that I have created the storage block directly in the Prefect web UI, and I'm loading it in my code, something like this:
Copy code
from prefect import flow
from prefect.filesystems import RemoteFileSystem
from prefect.serializers import JSONSerializer
from prefect.futures import as_completed
from prefect.task_runners import ThreadPoolTaskRunner

# Load the storage block you saved previously
r2_storage = RemoteFileSystem.load("r2-persistent-storage")

# Choose serializer for your payloads
json_ser = JSONSerializer()

@flow(
    log_prints=True,
    task_runner=ThreadPoolTaskRunner(max_workers=3),
    persist_result=True,
    result_storage=r2_storage,
    result_serializer=json_ser,
)
def my_flow_function():
    [...]
Another thing to note is that this code is working for already existing deployments, the error is only thrown when trying to deploy a new deployment. Any idea whats going on?
m
thought for 385.5 seconds
Thanks for the detailed context — this is a classic “environment mismatch at import time” issue. What’s happening - In Prefect 3.x, when you pass a Block object to
result_storage
in the flow decorator, Prefect validates that it’s a server‑persisted block (it must have a
_block_document_id
). - You’re calling
RemoteFileSystem.load("r2-persistent-storage")
at module import time. Locally, that succeeds because your profile/API key/workspace match the UI where you created the block. - During
prefect deploy
, your code is imported in a different environment/profile/workspace context. The
.load(...)
call does not resolve to a persisted block in that context, so Prefect sees an object without a server ID and raises: TypeError('Result storage configuration must be persisted server-side. Please call
.save()
on your block before passing it in.') Why it only happens on new deployments - Existing deployments were created when your environment resolved the block correctly; new deployments import your code in the current deploy environment which likely doesn’t share the same profile/workspace or API credentials. Two good fixes 1) Recommended: pass the block as a string (name or slug) instead of loading it at import time. This defers resolution until runtime in the correct environment:
Copy code
from prefect import flow
from prefect.serializers import JSONSerializer
from prefect.task_runners import ThreadPoolTaskRunner

@flow(
    log_prints=True,
    task_runner=ThreadPoolTaskRunner(max_workers=3),
    persist_result=True,
    result_storage="remote-file-system/r2-persistent-storage",  # or just "r2-persistent-storage"
    result_serializer=JSONSerializer(),
)
def my_flow_function():
    ...
Accepted forms: - Block name:
"r2-persistent-storage"
- Block slug:
"remote-file-system/r2-persistent-storage"
2) Ensure the deploy environment is pointed at the same workspace/profile where the block exists, then keep your current pattern. Before running
prefect deploy
, do: - Verify/set active profile:
Copy code
prefect profile ls
  prefect profile use <your-profile>
- Verify/set workspace:
Copy code
prefect cloud workspace ls
  prefect cloud workspace set <org>/<workspace>
- Make sure
PREFECT_API_KEY
is set for that profile. Practical troubleshooting - In the same shell where you run
prefect deploy
, try a quick import to confirm the block resolves:
Copy code
python -c "from prefect.filesystems import RemoteFileSystem; print(RemoteFileSystem.load('r2-persistent-storage')._block_document_id)"
If that fails or prints
None
, the block isn’t found in the active workspace/profile. - Double-check the block lives in the exact workspace you’re deploying to (blocks are workspace-scoped). References - Results config and block resolution in Prefect 3.x: Results - Settings, profiles, and workspaces: Settings &amp; Profiles, Workspaces If you switch
result_storage
to the string reference and ensure your deploy shell points at the right workspace/profile, this error should go away. Want me to look at your exact
prefect deploy
command and profile/workspace output to confirm?
b
@Marvin why is it recommended to pass the block as a string (name or slug) instead of loading it at import time, especially as the Prefect UI clearly suggests using the following code to call load the block?
Copy code
from prefect.filesystems import RemoteFileSystem

remote_file_system_block = RemoteFileSystem.load("r2-persistent-storage")
I have run the
prefect profile
command to ensure I am connected to my instance:
Copy code
prefect profile use personal
⠹ Checking API connectivity...
Connected to Prefect server using profile 'personal'
but still i get this warning that the block 'Result storage configuration must be persisted server-side.'
m
thought for 79.4 seconds
Great question — and I get why it’s confusing when the UI shows a
.load(...)
snippet. Why passing a string is recommended for result_storage - Import-time vs runtime:
prefect deploy
imports your module to build the deployment. When you call
RemoteFileSystem.load(...)
at the top of your module, that API call happens during import. If the deploy environment’s profile/workspace is different (or just not initialized yet), the loaded object won’t have a server-side ID and Prefect raises:
Copy code
TypeError("Result storage configuration must be persisted server-side. Please call `.save()` on your block before passing it in.")
- Deferred resolution: If you pass a string (block name or slug) instead:
Copy code
result_storage="remote-file-system/r2-persistent-storage"
Prefect defers the actual block lookup to runtime (when the flow runs), using the active API context then. This avoids import-time API calls and is much more robust across environments, CI/CD, profiles, and workspaces. - Security/portability: Passing a string avoids embedding a concrete Block object (possibly including secrets) into your imported code at deploy time. Why the UI shows
.load(...)
- The UI snippet is a generic example for using a block in code. It’s totally fine inside a flow/task at runtime. - For decorator configuration (like
result_storage
), the “string reference” pattern is safer because the decorator runs at import time. How to fix your flow quickly Change this:
Copy code
r2_storage = RemoteFileSystem.load("r2-persistent-storage")

@flow(
    persist_result=True,
    result_storage=r2_storage,
    ...
)
To this:
Copy code
@flow(
    persist_result=True,
    result_storage="remote-file-system/r2-persistent-storage",  # or "r2-persistent-storage"
    ...
)
Why you still see the “persisted server-side” error after
prefect profile use personal
- “Connected to Prefect server using profile 'personal'” only confirms connectivity. It doesn’t guarantee the block exists in that workspace or that the import-time
.load(...)
resolves to a persisted block with an ID. - The error is raised when the Flow is constructed because Prefect checks: if you pass a Block object (not a string), it must have a
_block_document_id
. If
.load(...)
didn’t resolve against your server/workspace at import time, that attribute is missing and you get the error. Verify the block in the same shell you run deploy - Confirm the block exists in this workspace:
Copy code
prefect block ls
prefect block inspect remote-file-system/r2-persistent-storage
- Sanity check the ID at import time:
Copy code
python - <<'PY'
from prefect.filesystems import RemoteFileSystem
b = RemoteFileSystem.load("r2-persistent-storage")
print("block_document_id:", getattr(b, "_block_document_id", None))
PY
If that prints
None
or raises, your deploy shell is not pointed at the workspace where the block exists. Fix by setting your workspace or API settings before deploy: - Cloud:
Copy code
prefect cloud workspace ls
  prefect cloud workspace set <org>/<workspace>
- Server: Ensure
PREFECT_API_URL
points to your server and your profile has the correct settings. Bottom line - For
result_storage
specifically, prefer passing the block by string (name or slug). It’s resilient and avoids import-time API lookups. - If you prefer to keep
.load
, you must ensure the exact workspace/profile is active in the same shell where you run
prefect deploy
and that the loaded object has a
_block_document_id
. If you share: -
prefect version
- Output of
prefect block inspect remote-file-system/r2-persistent-storage
- The exact
prefect deploy
command you’re running …I can pinpoint the mismatch quickly. Here are relevant docs if you want to dig deeper: - Results: Results - Blocks: Blocks - Profiles & settings: Settings &amp; profiles