AP
04/21/2026, 9:47 PMMarvin
04/21/2026, 9:47 PMMarvin
04/21/2026, 9:47 PMAP
04/21/2026, 9:48 PMMarvin
04/21/2026, 9:49 PMMarvin
04/21/2026, 9:57 PM# One-time setup (in a script or UI)
from prefect.filesystems import RemoteFileSystem
RemoteFileSystem(basepath="<s3://my-bucket/shared/>").save("shared-store", overwrite=True)
DataGenerationAgent writes a predictable key:
from prefect import flow
from prefect.results import ResultStore
from prefect.filesystems import RemoteFileSystem
@flow
def data_generation(batch_id: str) -> str:
store = ResultStore(result_storage=RemoteFileSystem.load("shared-store"))
key = f"data/{batch_id}.pickle"
payload = {"rows": [1, 2, 3]} # your generated data
store.write(obj=payload, key=key)
return key # return or log the URI/key for downstream
AnalysisAgent reads that key:
from prefect import flow
from prefect.results import ResultStore
from prefect.filesystems import RemoteFileSystem
@flow
def analysis(batch_id: str):
store = ResultStore(result_storage=RemoteFileSystem.load("shared-store"))
record = store.read(key=f"data/{batch_id}.pickle")
data = record.result
# analyze `data` here
- Docs: Results (Prefect 3)
- Blocks overview: Blocks
- Integrations (S3/GCS/Azure, etc.): Integrations
2) Programmatic chaining with parameters
- Trigger AnalysisAgent from DataGenerationAgent and pass a pointer (e.g., S3 key) rather than the data itself.
from prefect import flow
from prefect.deployments import run_deployment
@flow
def orchestrator(batch_id: str):
# generate data first (optionally returns the key)
run_deployment(name="data-generation/deploy", parameters={"batch_id": batch_id})
# then kick off analysis with the same batch_id (or the returned key)
run_deployment(name="analysis/deploy", parameters={"batch_id": batch_id})
- Docs: Run deployments from flows
3) Event-driven chaining (decoupled)
- Use an Automation to run AnalysisAgent when DataGenerationAgent completes; pass parameters via templates.
from prefect import flow, serve
from prefect.events import DeploymentEventTrigger
@flow
def analysis(batch_id: str):
...
if __name__ == "__main__":
analysis_dep = analysis.to_deployment(
name="analysis",
triggers=[
DeploymentEventTrigger(
expect={"prefect.flow-run.Completed"},
match_related={"prefect.resource.name": "data-generation"},
parameters={
"batch_id": {
"__prefect_kind": "jinja",
"template": "{{ flow_run.parameters['batch_id'] }}"
}
}
)
]
)
serve(analysis_dep)
- Docs: Chain deployments with events
- Passing event data/parameters: Pass event payloads to flows
4) Prefect Artifacts (for small/medium results and UI display)
- Good for summaries, tables, markdown, images. Stored in Prefect’s DB and retrievable by key; not ideal for large binary datasets.
- Docs: Artifacts
Recommendations
- Large data: store in S3/GCS/Azure and pass a URI/key; use ResultStore or direct SDKs (boto3/gcsfs/azure) for full control.Marvin
04/21/2026, 9:57 PMrun_deployment (programmatic).
- Human-friendly outputs (reports/metrics): create artifacts alongside your data handoff.
Quick clarifiers to tailor the best pattern:
- About how large are the handoff files?
- Do you have a preferred storage (S3, GCS, Azure, NFS)?
- Should AnalysisAgent run automatically after generation, or on a schedule?