Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Prefect 1.0

Is it bad practice to pass large amounts of data via Parameters?

I have a Flow `A` I'm using as a flow-of-flows. `A` goes and downloads a bunch of data from S3. I want to pass this data (say, 500MB of JSON) to a child flow `B` via Parameter.

Yes definitely that data would be written to our database and needs to go over the network

I presume that would exceed our limits and be denied. I’d pass the data via a reference to the S3 file instead.

Oh, well -- we aren't using Prefect cloud.

Would the same thing apply with our self-managed prefect server  -- parameters are written to the postgres DB ?

yeah, we have 1.0 self managed. `parameters` is a column of the `flow_run` table and includes the actual values