Great question — there isn’t a single “best” object type. The right choice depends on how your tasks/flows run and whether you persist results. Since you’re on Prefect 2 (Cloud) and GCP, here’s a practical guide: Short answer - For small/medium payloads: plain dicts/lists/primitive types are the safest and most portable. - For structured configs: dataclasses or Pydantic models are great — pick one based on whether you want validation (Pydantic) or light weight (dataclasses). - For large dataframes/arrays/blobs: store them in GCS and pass a path/URI, not the data itself. Why this matters in Prefect 2.x - In-memory handoff: Within the same process, most Python objects just work. - Across processes/workers (e.g., Dask/Ray) or when persisting results, objects must be serializable. Prefect/Dask/Ray rely on cloudpickle for complex objects; JSON is best for portability. - Persisted results: In 2.x, nothing is persisted by default. If you set

persist_result=True

, you should also choose a

result_serializer

and, for remote storage, a

result_storage

(e.g., a GCS block). Recommendations - Use dicts/lists for most hand-offs, especially if you want JSON serialization. - Use dataclasses when you want type-safety without overhead. If you persist with JSON, return `asdict(my_dc)`; otherwise use pickle. - Use Pydantic models if you want validation. If you persist with JSON, return `model.dict()`; otherwise use pickle. - Avoid passing open connections/clients/sessions; recreate these in each task or use blocks. - For large objects (dataframes, model weights, binary data): - Write to GCS in the producer task and return the blob path. - Read from that path in the consumer task. - This avoids big in-memory payloads and costly (or fragile) serialization. Concrete examples 1) Dataclass + pickle (simple and reliable)

from dataclasses import dataclass
from prefect import task, flow
from prefect.serializers import PickleSerializer

@dataclass
class Config:
    project: str
    threshold: float

@task(persist_result=True, result_serializer=PickleSerializer())
def make_config():
    return Config(project="my-proj", threshold=0.8)

@task
def use_config(cfg: Config):
    return cfg.threshold * 2

@flow
def main():
    cfg = make_config()
    return use_config(cfg)

2) Pydantic + JSON (structured, portable)

from pydantic import BaseModel  # v1 in most Prefect 2 envs
from prefect import task, flow
from prefect.serializers import JSONSerializer

class Record(BaseModel):
    id: int
    name: str

@task(persist_result=True, result_serializer=JSONSerializer())
def make_record() -> dict:
    # return a JSON-serializable shape
    return Record(id=1, name="alpha").dict()

@task
def use_record(rec: dict):
    return rec["id"]

@flow
def main():
    rec = make_record()
    return use_record(rec)

3) Large data via GCS (best for big payloads on GCP) - Pattern: write to GCS in producer, pass the blob path, read in consumer. - Use the

GcsBucket

block in

prefect-gcp

for uploads/downloads. Example patterns and methods are documented here: - Prefect 2: GCS integration (GcsBucket) - This avoids pickling large objects and keeps results fast and stable across infrastructure boundaries. Gotchas to avoid - Don’t pass DB connections, open files, or non-picklable objects between tasks. - If you want JSON persistence, don’t return raw dataclass/Pydantic instances — return `asdict()`/`.dict()` instead. - Pickle is fast and flexible but not portable across languages and should only be used with trusted data sources. Useful docs - Results, persistence, and serializers (Prefect 2): Results in Prefect 2 - GCS integration (Prefect 2): GCS block and helpers If you can share: - Are you using a task runner like Dask/Ray?

- Do you plan to persist results, and if so, where (GCS)? - Typical payload sizes you’re passing? I can tailor a recommendation (and code) for your setup.

Short answer: pass the PrefectFuture, not

.result()

, whenever you’re wiring tasks together. Why: - Keeps the dependency graph intact so the UI shows lineage and Prefect can propagate retries/skips/cancellations automatically - Preserves concurrency; calling

.result()

blocks the flow and can serialize large data through the flow process unnecessarily - Plays better with persisted results; the downstream task will fetch the value where it runs instead of materializing it in the flow then re-sending it Use

.result()

only when you actually need the Python value in the flow code (e.g., conditional branching, logging, or returning a final value). If you only need to wait on completion without loading the data, prefer:

f.wait()  # waits for completion, doesn't materialize the value

Example:

a = task_a.submit()
b = task_b.submit(a)          # recommended: pass the future
# vs
b = task_b.submit(a.result()) # blocks, loses explicit upstream link

If you’re moving very large objects between tasks (common on GCP when using distributed runners), consider persisting them (e.g., GCS) and passing references instead. Happy to suggest a pattern based on how you’re running (threads/processes/Dask/Ray) and the object sizes.

Great question. In Prefect 2, here’s the rule of thumb I recommend: Use tasks by default - Best for fine-grained, fast, parallelizable work within a single flow run - Low overhead; great for fan-out/fan-in patterns - Fine control of retries/timeouts per unit of work - Results and caching work naturally at the task level Promote to a subflow when the step is an orchestration boundary - You want the step to be runnable/schedulable on its own (it can have its own deployment) - You want to retry/fail a group of tasks as a unit - You want clearer visibility and isolation in the UI for a major phase (e.g., extract/transform/load) - You plan to re-use the step across multiple parent flows - You may trigger the step on a different work pool/infrastructure (via

run_deployment

) Direct subflow call vs run_deployment - Call a subflow function directly: executes in the same process/infrastructure as the parent, but creates its own flow run record and logs. Lowest overhead, great for logical grouping and per-phase retries. - Use `run_deployment(...)`: spins up a separate flow run picked up by workers for that deployment’s work pool. Best when you need different infra (e.g., GKE for heavy compute, Cloud Run for bursts) or decouple lifecycles. Pass small parameters and share large data via storage (e.g., GCS URIs). Example structure

from prefect import flow, task
from prefect.deployments import run_deployment

@task(retries=2)
def transform(chunk): ...

@flow
def transform_phase(chunks):
    futures = [transform.submit(c) for c in chunks]
    return [f.result() for f in futures]

@flow
def etl_main():
    chunks = ...  # prepare data
    # In-process subflow for grouping + visibility
    results = transform_phase(chunks)

    # Or dispatch to separate infra if needed
    # run_deployment(name="transform-phase/gke", parameters={"chunks": chunk_refs})

A few practical tips - Keep subflows “coarse”: a handful of meaningful phases beats dozens of tiny subflows - Keep tasks “small”: fast, composable units; parallelize with

.submit

- Large payloads between subflows/deployments: write to GCS and pass references instead of big objects - If a failure in a phase should cause the whole phase to retry or fail together, make it a subflow If you can share your pipeline shape, data sizes, and where you want to run things on GCP (e.g., GKE vs Cloud Run vs local), I can recommend a concrete layout.

Short answer: “big” depends on the boundary you’re crossing. Here are practical thresholds for Prefect 2 on Cloud: - Flow/deployment parameters (over the API) - Treat anything >100–500 KB as big. Aim to keep parameters under ~100 KB. - Above ~1 MB, you’re likely to hit slow requests, timeouts, or DB bloat. Prefer writing to GCS and passing a URI. - Task → task in the same flow run/process (passing PrefectFutures) - Bound by your worker’s memory. A single object >50–100 MB is “big” because parallelism multiplies memory pressure and serialization cost. - Hundreds of MB to GB-sized Python objects can crash the run or thrash memory when you fan out. - Across infrastructure boundaries (e.g., parent flow uses run_deployment to start another flow) - Only pass small scalars/metadata. Treat >100 KB as big and pass a storage reference instead. - Logs/metadata - Keep individual payloads small (tens of KB). Large strings get slow and may be truncated. Recommended pattern for “big” data - Write the data to GCS and pass lightweight references (URIs, keys, or metadata). This avoids pushing bytes through the API or keeping huge objects in memory across many futures. Example with GCS (Prefect 2 + prefect-gcp)

from prefect import flow, task
from prefect_gcp.cloud_storage import GcsBucket

@task
def write_big(df) -> str:
    gcs = GcsBucket.load("my-gcs")
    path = f"data/run-{prefect.runtime.flow_run.id}.parquet"
    # Save df to local tmp then upload, or serialize bytes and use upload_from_file
    local = "/tmp/data.parquet"
    df.to_parquet(local)
    gcs.upload_from_path(from_path=local, to_path=path)
    return f"gs://{gcs.bucket}/{path}"

@task
def read_big(uri: str):
    # Download and read back
    gcs = GcsBucket.load("my-gcs")
    path = uri.replace(f"gs://{gcs.bucket}/", "")
    local = "/tmp/in.parquet"
    gcs.download_object_to_path(object_name=path, local_path=local)
    import pandas as pd
    return pd.read_parquet(local)

@flow
def pipeline():
    # Produce big data and pass only a URI between tasks
    uri = write_big.submit(prepare_df()).result()
    result = read_big.submit(uri)
    return result.result()

Rules of thumb - If it must cross the API (parameters to another flow/deployment), keep it under ~100 KB and avoid >1 MB. - If it stays within one flow run and you’re fanning out, keep each object under ~50–100 MB to protect memory. - For anything larger, persist to storage (GCS) and pass references. If you can share what the object is (e.g., pandas DataFrame), typical size, and whether you’re passing it task→task or flow→deployment, I can give a more precise recommendation and a tailored example for GCP. Further reading: - Task runners and concurrency in Prefect 2 - Scaling patterns and mapping considerations