<@ULVA73B9P> Why didn't you answer on direct messa...
# marvin-ai
a
@Marvin Why didn't you answer on direct message?
m
thought for 31.7 seconds
Sorry about that! I don’t see or get notified about DMs unless: - I’ve been added to the DM (apps must be invited) - You started the DM with me directly or mentioned @Marvin to invite me - Your workspace allows app DMs If you’d like, just drop your question here and I’ll help right away. Or, in the DM: 1) Add me to the conversation (Add people/apps or mention @Marvin) 2) Send the message again so I can see it What can I help you with now?
a
@Marvin I'm completely new to Prefect. I using it with python. I started deploying. Now I have two independent deploys and I'm wondering which mechanisms I have to share data between them. Example: 1. DataGenerationAgent 2. AnalysisAgent the one generates data, the other analyzes data. How to achieve that? I need somehow a shared storage backend.
m
thought for 476.4 seconds
Great question — there are a few solid patterns in Prefect 3.x for handing off data between independent deployments. Pick the one that fits your data size and coupling needs: 1) Shared storage (most common) - Write outputs from DataGenerationAgent to a shared filesystem (S3/GCS/Azure/SMB/local path) and have AnalysisAgent read them. - You can use Prefect’s ResultStore for simple, serialized handoffs with predictable keys. Minimal example with a shared RemoteFileSystem block:
Copy code
# One-time setup (in a script or UI)
from prefect.filesystems import RemoteFileSystem
RemoteFileSystem(basepath="<s3://my-bucket/shared/>").save("shared-store", overwrite=True)
DataGenerationAgent writes a predictable key:
Copy code
from prefect import flow
from prefect.results import ResultStore
from prefect.filesystems import RemoteFileSystem

@flow
def data_generation(batch_id: str) -> str:
    store = ResultStore(result_storage=RemoteFileSystem.load("shared-store"))
    key = f"data/{batch_id}.pickle"
    payload = {"rows": [1, 2, 3]}  # your generated data
    store.write(obj=payload, key=key)
    return key  # return or log the URI/key for downstream
AnalysisAgent reads that key:
Copy code
from prefect import flow
from prefect.results import ResultStore
from prefect.filesystems import RemoteFileSystem

@flow
def analysis(batch_id: str):
    store = ResultStore(result_storage=RemoteFileSystem.load("shared-store"))
    record = store.read(key=f"data/{batch_id}.pickle")
    data = record.result
    # analyze `data` here
- Docs: Results (Prefect 3) - Blocks overview: Blocks - Integrations (S3/GCS/Azure, etc.): Integrations 2) Programmatic chaining with parameters - Trigger AnalysisAgent from DataGenerationAgent and pass a pointer (e.g., S3 key) rather than the data itself.
Copy code
from prefect import flow
from prefect.deployments import run_deployment

@flow
def orchestrator(batch_id: str):
    # generate data first (optionally returns the key)
    run_deployment(name="data-generation/deploy", parameters={"batch_id": batch_id})
    # then kick off analysis with the same batch_id (or the returned key)
    run_deployment(name="analysis/deploy", parameters={"batch_id": batch_id})
- Docs: Run deployments from flows 3) Event-driven chaining (decoupled) - Use an Automation to run AnalysisAgent when DataGenerationAgent completes; pass parameters via templates.
Copy code
from prefect import flow, serve
from prefect.events import DeploymentEventTrigger

@flow
def analysis(batch_id: str):
    ...

if __name__ == "__main__":
    analysis_dep = analysis.to_deployment(
        name="analysis",
        triggers=[
            DeploymentEventTrigger(
                expect={"prefect.flow-run.Completed"},
                match_related={"prefect.resource.name": "data-generation"},
                parameters={
                    "batch_id": {
                        "__prefect_kind": "jinja",
                        "template": "{{ flow_run.parameters['batch_id'] }}"
                    }
                }
            )
        ]
    )
    serve(analysis_dep)
- Docs: Chain deployments with events - Passing event data/parameters: Pass event payloads to flows 4) Prefect Artifacts (for small/medium results and UI display) - Good for summaries, tables, markdown, images. Stored in Prefect’s DB and retrievable by key; not ideal for large binary datasets. - Docs: Artifacts Recommendations - Large data: store in S3/GCS/Azure and pass a URI/key; use ResultStore or direct SDKs (boto3/gcsfs/azure) for full control.
- Need auto-run of AnalysisAgent: use Automations (event-driven) or
run_deployment
(programmatic). - Human-friendly outputs (reports/metrics): create artifacts alongside your data handoff. Quick clarifiers to tailor the best pattern: - About how large are the handoff files? - Do you have a preferred storage (S3, GCS, Azure, NFS)? - Should AnalysisAgent run automatically after generation, or on a schedule?