https://prefect.io logo
Title
a

Anders Segerberg

05/02/2023, 3:21 PM
Prefect 1.0 Is it bad practice to pass large amounts of data via Parameters? I have a Flow
A
I'm using as a flow-of-flows.
A
goes and downloads a bunch of data from S3. I want to pass this data (say, 500MB of JSON) to a child flow
B
via Parameter.
z

Zanie

05/02/2023, 3:23 PM
Yes definitely that data would be written to our database and needs to go over the network
I presume that would exceed our limits and be denied. I’d pass the data via a reference to the S3 file instead.
a

Anders Segerberg

05/02/2023, 3:50 PM
thank you for the context
Oh, well -- we aren't using Prefect cloud.
Would the same thing apply with our self-managed prefect server -- parameters are written to the postgres DB ?
k

Kyle McChesney

05/02/2023, 5:12 PM
yeah, we have 1.0 self managed.
parameters
is a column of the
flow_run
table and includes the actual values
a

Anders Segerberg

05/02/2023, 5:15 PM
thank you!